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Foreword 


In 2005 the Portuguese government invited ENQA to appoint a panel of international 
experts with two interrelated tasks. First, to review the existing Portuguese quality 
assurance practices as conducted by the Portuguese National Council for the Evaluation 
of Higher Education, CNAVES. Secondly, to provide recommendations to the 
Portuguese government on the organisation, processes and methods of establishment 
of a national accreditation system which would meet the European Standards and 
Guidelines for Quality Assurance in the European Higher Education Area. 

The Board of ENQA accepted this invitation and appointed an international review 
panel. 

The following report presents the results of the panel’s work on its double task. It is 
our hope that the conclusions and recommendations of the panel will serve not only as 
a platform for the future Portuguese quality assurance system, but also as an inspiration 
and benchmark for other national quality assurance systems. 


Peter Williams 

ENQA President 


Christian Thune 

Chair of the review panel 


1. Executive summary 

The purpose of this report is to review the past and present external quality assurance 
for higher education in Portugal and to recommend a future quality assurance and 
accreditation system that meets the standards for quality assurance in the European 
Higher Education Area. The review was carried out on the initiative of the Portuguese 
Ministry of Science, Technology and Higher Education (MCTES) which requested 
ENQA to appoint an international panel to conduct the review. 

During the review process, the Portuguese government announced that the present 
evaluation system will be dismantled by the end of 2006 and replaced by a new 
system for accreditation by the beginning of 2007. This initiative parallels activities 
in a growing number of European countries that are applying quality assurance 
systems for higher education that are based on an accreditation approach. The panel 
is, however, convinced that the new system would benefit from a broader quality 
assurance approach than one based exclusively on accreditation. 

The panel has completed a review of the past and present system in accordance with 
the agreement with MCTES, and the European standards for the review of external 
quality assurance agencies have formed the basis for the review process. Considering 
the fact that the European standards were only recently adopted by the relevant 
ministers of the Bologna signatory countries, the current quality assurance agency, 
the National Council for Evaluation of Higher Education (CNAVES), cannot in all 
fairness be expected to have developed its organisation and processes according to those 
standards. However, it has been the conclusion of the panel that the review would 
suffer from a serious lack of transparency and comparability if the standards were not 
applied as the frame of reference for the review. 

Overall conclusions 

Quality assurance of higher education in Portugal has been based on the Higher 
Education Evaluation Act 1994. This act was closely developed in cooperation with the 
Conference of Rectors of Public Universities (CRUP) and forms the backbone of what 
is widely termed “the contractual model”. The act resulted in the establishment of 
CNAVES as a national agency for quality assurance of higher education. 

The act was highly inspired by the Dutch model and retains many characteristics 
from this, including, above all, placing responsibility for the quality assurance system 
with an organisation representing the institutions of higher education. 

The contractual model, which reflects the large extent of institutional representation 
through the representative entities and evaluation councils, has created a strong 
sense of ownership and commitment to the quality assurance processes, which would 
otherwise have been extremely difficult to achieve. The model was fit for purpose at the 
time of its establishment, but was from the early days also associated with a number of 
weaknesses, which are analysed in this review. 

Ten years of CNAVES’s evaluations have, however, also accumulated a number of 
positive experiences that should be considered carried over into a new quality assurance 
system. 

The major strengths and weaknesses of the past and present quality assurance system 
identified by the panel are summarised in the following. 


The major strengths include: 

Establishment of a self-evaluation culture 

The evaluation processes have contributed to the build up of a self-evaluation 
culture at many higher education institutions. A number of institutions have 
developed their own internal quality assurance systems as a result of CNAVES’s 
evaluations. The evaluations have further assisted many institutions in becoming 
more reflective about their practice, and the institutions have made their 
education programmes more transparent to the public and students. The panel 
therefore believes that the future quality assurance system should maintain a 
strong focus on quality improvement. 

Appropriate methods 

The methodological model applied by CNAVES is in principle appropriate and 
in many respects in compliance with European standards and practice. The 
model contains most characteristics of other European models and includes self- 
evaluation, external review by external experts, a site visit and a published report. 
The model should, therefore, in outline continue in the future quality assurance 
system. 

Assurance of a more equal representation of sub-sectors 
The existing quality assurance system is comprehensive and includes all higher 
education institutions; public/private and universities/polytechnics. The inclusion 
of the private sector in CNAVES’s evaluations has helped improve the recognition 
and reputation of the sector. To some extent, the division into four evaluation 
councils has contributed to a more equal representation between the sub sectors 
in the quality assurance processes, though this is not fully realised. Although 
the panel strongly recommends a comprehensive system for the future, the 
new quality assurance system should also be able to reflect the present divided 
structure. 

The major weaknesses include: 

Limited independence 

The European standards emphasise that agencies must be independent from 
ministries as well as from higher education institutions. The representative 
nature of CNAVES and the evaluation councils, with the strong domination of 
particularly the public universities, compromises the independence of the system. 

The involvement of representatives of the public universities in the design 
process indicates clearly that the methods and procedures were not defined and 
developed in a manner that was sufficiently autonomous and independent of the 
higher education institutions. 

Furthermore the procedures for the appointment and nomination of external 
experts are not sufficiently independent of either the institutions or the Ministry. 

Lack of sufficient operational efficiency and consistency 
The two-tier organisational structure that includes CNAVES as the meta-organ 
and the evaluation councils as the operational organisations is complex and has 
created an organisation that suffers from a certain level of operational inefficiency 


and inconsistency. The many levels in the structure are time-consuming and 
result in an unduly complicated system detracting from the quality improvement 
purpose. 

The limited staff capacity at CNAVES and the evaluation councils has 
reinforced these weaknesses. As a consequence of the two-tier structure, the 
technical and administrative support is fragmented in several secretariats. The 
few support staff in CNAVES and in each council do not have the capacity to 
provide a proper quality control of the processes. The consequence is a limited 
degree of consistency in evaluation processes and in the final reports prepared by 
the various expert teams. 

Lack of consequences 

A crucial deficiency of the existing quality assurance system is the lack of 
consequences or follow-up on evaluations. The distinct lack of follow-up is due 
to a number of factors. The passiveness on the part of the Government is a 
major part of the explanation as is the lack of commitment from higher education 
institutions. But CNAVES’s lack of activity does also bear part of the responsibility. 
Follow-up visits to discuss reviews and to explore what institutional actions have 
been taken in response to the reviews could also have been undertaken by 
CNAVES itself, and this would have been consistent with the professed goal 
of self-improvement. Additionally, the fact that the reports have often been 
formulated in a vague manner, without clear conclusions and recommendations, 
has further constrained the possibility for follow-up. 

The evaluations are cyclical, with a first round from 1995-2000 and a second 
round from 2000-2005. The cyclical basis had in principle provided some sort 
of feed-back on the follow-up to the previous evaluations, but this has evidently 
not been the case. The evaluations in the second round did not cover or reflect 
the results from the first round. On the other hand it should be added that a 
third cycle was in the planning that would have addressed several key problems 
identified during the first two cycles. 

Reliance on national experts 

The reliance on national Portuguese experts poses a problem. The pool of 
Portuguese experts is limited and, as is generally the case with academic 
communities in small countries, there is often considerable familiarity between 
reviewers and the reviewed. This combination potentially compromises the 
necessary independence and objectivity of the external evaluations. Secondly, the 
dominant use of national experts has limited inspiration from experts with an 
international background and perspective. 

Major recommendations 

On the basis of the ENQA panel’s review of the experience with the present system, 
the panel agrees that a system change is now due regarding quality assurance of 
Portuguese higher education. The recommendations in the report specify the panel’s 
view on the direction and specific content of these changes. The basic change is the 
introduction of a new accreditation system. The ENQA panel defines the purpose 
of the new accreditation system to be partly the obvious one of assuring that 


quality at higher education institutions meets the requirement of the government and 
stakeholders and provides institutions with benchmarks on which to build their further 
development. But the panel wishes to emphasise that accreditation must also make 
the quality of Portuguese higher education programmes transparent and comparable in 
an international context. Only in this way can Portuguese higher education secure its 
place in the global context of higher education, and thereby the necessary international 
mobility for staff and students. 

Higher education has, namely in a global context, begun to take on some of the 
characteristics of a market place, and the new accreditation system must be designed to 
support efforts to attract the best staff and students from abroad and to provide access 
for Portuguese staff and students in academically attractive institutions abroad. 

Therefore, the starting point for the ENQA panel is that the coming quality 
assurance system must comply with the code of practice expressed in the European 
standards. On this overall background, the panel recommends to: 

Combine accreditation with academic audit at institutional level 
The decision of the Portuguese government to establish a new agency for 
accreditation should not preclude a combination of programme accreditation with, 
for instance, academic audit at institutional level. It is the opinion of the panel 
that a mature quality assurance system should operate at both programme and 
institutional level. 

A combination of different quality assurance methods will make it possible for 
the Portuguese government to include in its design for quality assurance both 
the element of control (accreditation) and the element of improvement (academic 
audit). 

Establish one strong independent national agency eor quality assurance 
The panel’s recommendation for a future quality assurance system is a single 
one-tier system. The agency should be responsible for accreditation as well as for 
audit processes. The tasks should be combined in one comprehensive agency to 
ensure a strong financial and staff base, and to benefit from the synergy between 
accreditation and academic audit. 

The agency must be truly independent of the government and higher education 
institutions. A small independent board with members appointed in their personal 
capacity should guarantee the independence of the agency. The board should be 
government appointed, but on the basis of a procedure that secures representation 
in relation to Portuguese society and higher education institutions. The board 
should be vested with the authority to make accreditation decisions. The 
membership of the board should reflect established professionalism in overseeing 
quality assurance processes. 

To ensure a wider involvement of relevant stakeholders in quality assurance, 
the board should be supplemented by an advisory council with representatives 
of relevant stakeholders, including higher education institutions, employer 
organisations and professional associations. 


Ensure operational independence of a new quality assurance agency 

THROUGH A LEGAL FRAMEWORK 

The operational independence of the agency should be rooted in official 
documents, e.g. a legislative act, which is the case in most European countries. 
The mandate of the new agency should be clearly described in the legal 
framework. At the same time, the legal framework should be formulated in a way 
that allows a certain degree of flexibility in the development and operation of the 
agency, e.g. determination of standards, adjustments in procedures, etc. 

Ensure professionalism and consistency in the work of the quality 

ASSURANCE AGENCY 

The credibility of external quality assurance processes depends on a high level 
of professionalism in the agency responsible. The level of credibility is closely 
related to the consistency between processes. Therefore, the European standards 
emphasise that quality assurance agencies should have adequate human and 
financial resources to match the tasks they are required to perform. 

The agency should have a strong, highly qualified staff with experience in 
quality assurance and with knowledge of the higher education system. The 
professional staff of the agency should carry a major obligation for the consistency 
of operational processes and reporting. 

This is especially important in relation to programme accreditations, which 
may otherwise suffer from the fact that different experts may view various 
elements differently and arrive at their own individual interpretations of the 
standards. 

Create a quality assurance system with clear consequences and well- 
defined PROCEDURES FOR FOLLOW-UP 

Consequences of accreditation and follow-up procedures in connection with 
academic audit should be clearly defined in the legal framework, and the agency 
should be responsible for providing the higher education institutions with proper 
information concerning the practical implications. 

An accredited programme should in the coming Portuguese system be ensured 
public funding and grants to the students. An accredited programme should also 
give the programme provider an added position of strength in the global market 
and thus attract researchers and students from abroad. Likewise, failure to obtain 
accreditation will involve a number of consequences - typically loss of financial 
grants or loss of the right to provide those specific programmes. 

The agency should be responsible for overseeing the follow-up on academic 
audits by carrying-out follow-up visit to discuss and explore what actions 
institutions have taken in response to a review. 

Use of international experts 

It is an element in the European standards that European external quality 
assurance systems should be encouraged to open their processes as much as 
possible to international expertise. In many European countries the inclusion of 
international experts is now being introduced. The new agency should ensure 
that international experts are invited as members of the expert panels in order to 


bring in an outside perspective and increase the credibility of the panels. The new 
agency should therefore itself be ensured sufficient funding to cover the added 
costs involved in basing quality assurance processes in a non-national language. 

About the recommendations 

The recommendations summarised above reflect the panel’s main priorities in 
relation to the establishment of a future quality assurance system. The report 
thus contains more recommendations than those summarised here. All of the 
recommendations will be presented in chapter seven. Furthermore, Appendix six 
contains a list of all recommendations. 


Abbreviations 


AACSB 

ABET 

ADISPOR 

ANET 

APESP 

CCISP 

CNAVES 

CRUP 

DGES 

EGA 

ENQA 

EHEA 

EU 

EUA 

EUR-AGE 

EQUIS 

EUP 

INQAAHE 

MCTES 

OE 

OECD 

UK 

US 

VSNU 


Association to Advance Collegiate Schools of Business 
Accreditation Board for Engineering and Technology 
Evaluation Council for Public Polytechnics 
Association for Technical Engineers 

Evaluation Council for Private Universities and Polytechnics 
Council for Polytechnics 

National Council for Evaluation of Higher Education 
Conference of Rectors of Public Universities 
Ministry of Science and Higher Education 
European Consortium for Accreditation 

European Association for Quality Assurance in Higher Education 

European Higher Education Area 

European Union 

European University Association 

Project for Accreditation of European Engineering Programmes 
European Quality Improvement System 
Evaluation Council for Public Universities 

International Network of Quality Assurance Agencies in Higher Education 
Ministry of Science, Technology and Higher Education 
Order of Engineers 

Organisation for Economic Cooperation and Development 
United Kingdom 
United States 

The Evaluation Agency of the Dutch Association of Universities 


2. Introduction 


The Portuguese government decided in 2004 to launch an extensive, independent 
and objective assessment of the Portuguese higher education system. The assessment 
should be conducted following international criteria and with the goal to guide the 
reorganisation and rationalisation of the system. 

This exercise was structured in three main tasks: 

• The Organisation for Economic Cooperation and Development (OECD) was 
commissioned to undertake the overall assessment of the Portuguese higher 
education system and its policies. 

• The European Association for Quality Assurance in Higher Education (ENQA) 
was invited to appoint a team of international experts to firstly review 

past and current Portuguese quality assurance practices as conducted by the 
Portuguese National Council for the Evaluation of Higher Education, CNAVES 
and including the related activities of “Representative Institutions”. Secondly 
to provide recommendations that could ensure the establishment of a national 
accreditation system meeting the European Standards and Guidelines for Quality 
Assurance in the European Higher Education Area. 

• In addition to these two tasks, an exercise of institutional evaluations was 
launched, which will become the ultimate responsibility of the higher education 
institutions themselves. The evaluations are run by the European University 
Association (EUA) in cooperation with the European Association of Institutions in 
Higher Education (EURASHE). 

2.1 Background 

MCTES drafted the terms of reference (see Appendix 1) and negotiated the contract 
with ENQA. In addition, the chair of the panel and the review secretary held a meeting 
with MCTES before commencing the review process to clarify outstanding issues 
and questions. MCTES was not involved in the review process as such, but provided 
logistical support in relation to the site visit. 

The European standards and guidelines for the review of external quality assurance 
agencies have formed the basis of the review process. The standards are summarised in 
Appendix five and can be read in their full length in the ENQA report “Standards and 
Guidelines for Quality Assurance in the European Higher Education Area” available at 
ENQA’s website www.enqa.eu. 

In accordance with the European standards, the key elements of the review process 
are: 

• a self-evaluation by the Portuguese National Council for the Evaluation of Higher 
Education (CNAVES) and its four evaluation councils, and six accounts by 
professional associations; 

• site visits by the ENQA panel; 

• a public review report. 

It is also in accordance with the European standards that the external ENQA panel 
has aimed to ensure that the review process is as transparent as possible, and that the 
process has allowed the institutions involved commitment and co-ownership. 


2.2 Mandate 

The review has a dual dimension. One task is to review the past and present evaluation 
system, and another task is to provide recommendations for a future accreditation 
system. Accordingly, the review provides an analysis of: 

• The existing Portuguese quality assurance practices as evident in the work of 
CNAVES and in the related activities and procedures of the four Evaluation 
Councils. 

• The review also takes into account the available knowledge of professional 
accreditation practices connected with professional educations, e.g. engineering 
and architecture. 

Eurthermore the report provides: 

• Recommendations that will lead to the establishment of a national quality 
assurance system and practice. The system and practice, based on programme 
accreditation and institutional audit, will meet the European Standards and 
Guidelines for Quality Assurance in the European Higher Education Area (EHEA), 
and, finally, will ensure appropriate academic and management structures for 
implementing adequate quality assurance and accreditation practice. 

The Portuguese government has announced that the present evaluation system will be 
dismantled by the end of 2006, and replaced by a new agency for accreditation by the 
beginning of 2007. Decisions on the design of the future accreditation system await the 
recommendations of this ENQA review. 

The ENQA panel has considered its mandate in the light of the terms of reference 
and recent developments concerning CNAVES. The panel does not consider these 
developments to imply any changes to the already agreed terms of reference for the 
review. Therefore, the panel has conducted the review of CNAVES according to the 
agreement with MCTES, and the panel’s views and conclusions based on the lessons 
learned from the CNAVES period are brought into the recommendations for a future 
system. 

The panel is convinced that this system will benefit from a broader quality assurance 
approach than one based exclusively on accreditation. Accordingly, the panel’s focus is 
on a new quality assurance system incorporating both accreditation and other more 
improvement-oriented approaches. 

2.3 Organisation 

The ENQA review has been carried out by a group of international experts appointed 
by the Board of ENQA. 

The members of the panel are: 

• Dr. Christian Thune, Chair, Executive Director of the Danish Evaluation 
Institute (EVA), past President of ENQA; 

• Professor David Dill, Professor of Public Policy at the University of North 
Carolina; 

• Professor Helmut Konrad, past member of the ENQA Board and past President 
of the Austrian Accreditation Council; 

• Professor Gemma Rauret, Executive Director of the Spanish Agency for 
Evaluation of Quality and Accreditation (ANECA); 


• Dr. Anette D0RGE Jessen, Review Secretary, Director of Projects at the Danish 
Evaluation Institute (EVA). 

Ms. Brit Buchhave, MA, Communication Consultant at the Danish Evaluation 
Institute (EVA), participated in the site visit and provided secretarial assistance to the 
panel. 

Further details regarding the panel members are provided in Appendix two. 

On 30 January 2006, the panel organized a launch seminar in Lisbon for CNAVES 
and the Evaluation Councils, where panel members Christian Thune, Anette Dorge 
Jessen and David Dill introduced the procedure and schedule for the review and the self 
evaluation manual for CNAVES. Later the same day, these panel members also held a 
meeting with representatives of the professional associations concerning their inclusion 
in the review process. 

The ENQA panel held a one-day meeting 1 May 2006 in Copenhagen to discuss 
the self-evaluation documents of CNAVES and the professional associations, and to 
prepare for the site visit in May 2006. Furthermore, the panel convened for a one-day 
meeting in 4 September 2006 in Copenhagen to discuss the draft report. Further 
communication between the panel members has primarily been via e-mail. 

15 September the panel sent CNAVES a first version of this report and invited 
comments on any factual errors or misinterpretations. Comments were also invited on 
the extent of the report’s fulfilment of the terms of reference for the review. CNAVES 
and the Evaluation Councils gave their comments in written statements. 23 October 
2006 the panel chair and the review secretary met in Lisbon with representatives of 
CNAVES and the Councils whose statements were discussed in an open and frank 
atmosphere. 

On the basis of the statements and the meeting 23 October the panel has made 
adjustments to the report as presented here in its final version. 

2.4 Documentation 

Three main types of documentation have formed the basis for the review. Firstly, 
the self-evaluation report prepared by CNAVES and the accounts prepared by six 
professional associations, including appendices; secondly, the site visit to Lisbon; and 
thirdly, the MCTES provided extensive documentation in English on the Portuguese 
higher education system. 

The views on the Portuguese system referred to throughout in the report are based 
on from the analysis of documentation by the ENQA panel. An important element of 
this documentation is the interviews with the stakeholders during the site visit 

Self-evaluation 

CNAVES conducted a self-evaluation, analysing the strengths and weaknesses of the 
past and present evaluation system and the evaluation procedures and processes. The 
self-evaluation was designed to fulfil two distinct aims: 

• to provide documentation to be used by the ENQA panel in relation to the site 
visits and preparation of the final report; 

• to stimulate discussions about strengths and weaknesses related to the focus areas 
of the review in order to ensure continuous improvement in the practice of the 
organisation. 


The self-evaluation was carried out in accordance with a set of guidelines outlined 
by the ENQA panel. The questions in the guidelines were formulated in order to assess 
CNAVES’ compliance with the European standards. The ENQA panel is conscious 
of the fact that the European standards were adopted by the relevant ministers of 
the Bologna signatory countries as late as May 2005, and that CNAVES cannot, 
therefore, in all fairness be expected to have developed its organisation and processes 
according to the standards. However, ENQA drafted the standards in accordance with 
generally agreed and shared best practice among European quality assurance agencies. 
Furthermore, it is an important dimension in the European standards that agencies 
must be regularly reviewed in terms of their compliance with the standards. 

Accordingly, it has been the conclusion of the ENQA panel that the review of 
CNAVES would suffer from a serious lack of transparency and comparability if the 
standards were not applied as the frame of reference for the review. The panel cannot 
of course expect full compliance of the present system with the standards, but it is 
the view of the panel that the standards may nevertheless function very adequately 
if interpreted as relative benchmarks for the quality and relevance of the Portuguese 
quality assurance system 

CNAVES and its four evaluation councils submitted their self-evaluation report at 
the end of April 2006. The ENQA panel recognizes the report as a valuable and 
comprehensive insight into the recent Portuguese quality assurance system. The report 
should be applauded for its satisfactory level of critical self-reflection and for its outline 
of the main strengths and weaknesses of the system and the key issues, from CNAVES ' 
point of view to be addressed by a future system. Many of the critical observations made 
by the ENQA panel in this report may also be identified in the CNAVES self-evaluation 
report itself. This testifies to the excellent work effort of the CNAVES self-evaluation 
group. It should Anally be mentioned that the report further benefits from the extensive 
and representative written documentation supplied as annexes. 

Accounts 

Six professional associations accepted the invitation to submit a brief account of their 
accreditation activities and plans. These accounts provide valuable knowledge of the 
experiences, competences, concerns and expectations of the professional associations 
towards a future accreditation system. The ENQA panel has especially noted the 
impressive account of the Professional Order of Engineers concerning their elaborate 
and extensive accreditation work. A list of the six professional associations is provided 
in Appendix three. 

Site visit 

The site visit took place between 14-19 May 2006. The programme for the visit was 
designed to ensure that both aspects of the review exercise were duly covered. The 
panel had drafted the programme and set up criteria for the selection of higher 
education institutions to be visited. In order to ensure that all relevant stakeholders 
were included, the MCTES had subsequently added a number of meetings for the 
approval of the ENQA panel. As a result the Anal programme was very comprehensive, 
but therefore also very rewarding in terms of the insights and understanding gained 
by the ENQA panel. 

See Appendix four for the programme for the site visit. 


2.5 Content of the report 

In addition to this introductory chapter 2, the report contains six chapters. 

Chapter 1 provides an executive summary of the main conclusions derived from 
the assessment of the existing quality assurance system, and a summary of the most 
important recommendations for the design of a future quality assurance system. 
Chapter 3 analyses the national context for higher education in Portugal. Chapter 
4 provides an analysis of the past and existing quality assurance system; chapter 5 
concerns the professional associations; chapter 6 outlines the conditions for a new 
quality assurance system; and chapter 7 provides specific recommendations for a new 
Portuguese system. 


3- The national context 

3.1 The national system for higher education 

In comparison to other EU countries, Portugal has a distinctive history and structure 
to its higher education system. An analysis of the existing and possible future quality 
assurance system in Portugal must, therefore, take into account these characteristics as 
well as the dynamic changes currently underway. 

The higher education system in Portugal is binary, comprising university institutions 
and polytechnic institutions, which can be public or private. There are more than 
r50 higher education institutions. The public sector includes 14 universities and a non- 
integrated university institution; r5 polytechnics and a number of polytechnic schools 
integrated in universities; 9 non-integrated nursing schools; 4 university-level military 
schools; and 5 polytechnic military schools. The private sector includes 34 university 
level institutions and no less than 66 polytechnics. The private sector also includes a 
Catholic university'. 

The university and polytechnic subsystems are mainly differentiated by their 
formative role in research, as is seen, for example, also in Denmark and the 
Netherlands. The polytechnics are vocationally or professionally oriented and do 
not carry out fundamental research as the universities do. Only applied research is 
conducted at the polytechnics. 

The number of higher education institutions expanded rapidly in the eighties and 
early nineties as a response to an increase in student enrolment. The number of 
students in higher education institutions has grown from 30,000 in the sixties to nearly 
400,000 in 2000^. The expansion of higher education was mainly seen in the private 
sector and particularly in the polytechnic subsystem, as the present high number of 
polytechnic institutions makes evident. 

The expansion of the higher education system is also reflected in a large number 
of undergraduate courses and study programmes. In 2004, the total number of 
undergraduate study programmes was r,y63, of which 647 were offered by public 
universities, 58 by the Catholic university, 299 by private universities, 532 by public 
polytechnics and 227 by private polytechnics^. 

A recent and projected downturn in enrolments has led to a marked imbalance 
between supply and demand for higher education study programmes. The result is 
enhanced competition across the higher education sector and has increased the need 
both for a more effective means of assuring academic standards as well as for more 
effective institutional management and governance. 

It is a shared view among stakeholders in Portugal that the uncontrolled expansion 
of the higher education system and, in particular, of the private sector has affected the 
quality of the education and compromised the credibility of the system. There was a 
national consensus in the early nineties to establish a national quality assurance system. 


^ Tertiary Education in Portugal; Background Report prepared to support the international assessment of the Portuguese system of 
tertiary education, April 2006 , page 106. 

^ Tertiary Education in Portugal; Background Report prepared to support the international assessment of the Portuguese system of 
tertiary education, April 2006 , page 9. 

^ Source: Self-evaluation report of CNAVES, April 2006 


to guarantee a minimum level of quality of existing study programmes. This system is 
the main object of this review. It can he said at this stage that the system was prohahly 
fit for purpose at the time of its establishment. However, from its early days the system 
was also associated with a number of weaknesses. Chapter four of the report includes a 
thorough analysis of the existing system’s strengths and weaknesses. 

3.2 Regulation of higher education 

The higher education system in Portugal is regulated through a number of degree 
laws'^. The common fundamental principle of the degree laws is the prevalent autonomy 
of the public universities. These universities are, in terms of the law, entitled to 
statutory, scientific, pedagogical, administrative and financial autonomy. This means 
that the universities have almost complete freedom to initiate, suspend or cancel study 
programmes. The Ministry of Science and Higher Education (DGES), which holds 
the formal responsibility for the universities, registers all new study programmes and 
degrees. With the University Autonomy Act of 1988, this approval process was reduced 
to a bureaucratic procedure without any actual impact. The Ministry is only entitled to 
refuse study programmes or degrees if they are inconsistent with formal requirements 
in terms of duration of the programme or number of total credits needed to obtain a 
degree. Regardless of this, decisions are not based on evaluations or accreditations. 

The autonomy of the public polytechnic institutions is almost similar to that 
of the universities, but the procedure for approval of new study programmes is 
somewhat more restrictive. The public polytechnics are not allowed to create new study 
programmes without a formal approval by the Ministry. They are obliged to submit all 
proposals to the Ministry (General Director for Higher Education at the Ministry of 
Science and Higher Education) for approval. The proposals are subjected to a technical 
review at the Ministry where the compliance with formal requirements in terms of 
duration and number of credits is tested. The review usually takes about a year. 

The private universities and the private polytechnics are regulated through the 
Private and Gooperative Higher Education Act. The institutions must be officially 
recognised in order to award degrees. The procedure for approval of new study 
programmes and degrees is much more restricted compared to the approval process 
applied to the public universities and polytechnics. The approval procedure can take up 
to four years and includes not only a ministerial technical review but also an academic 
review by an external expert committee. The prolonged duration of the approval 
process is due to the extended process for feed-back from the expert committees. 
However, there is no clear evidence that this extended procedure has improved the 
quality of new study programmes provided by the private sector, but it has undoubtedly 
created a more self-disciplinary behaviour in terms of number of applications for 
approval of new study programmes. This assumption is substantiated by the relatively 
low number of undergraduate study programmes offered by private institutions 
compared with the number of undergraduate study programmes offered at public 
institutions, even though the number of private institutions is substantially higher (see 
previous section). 
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The comprehensive act for the education system (act 46/86 of 14 October); The University Autonomy Act (act 108/88 of 24 
September); the Autonomy of Polytechnic Higher Education Institutions Act (act 54/90 of 5 September); the Private and 
Cooperative Higher Education Act; etc. (see page lo-ii in CNAVES Self-Evaluation report). 


The actual number of new degree study programmes registered at DGES illustrates 
the difference in the approval procedures between the private and public sector. 

In 2005, a total of 141 new degree study programmes for public universities were 
registered. In comparison, 61 of a total of 157 degree programme applications from 
private institutions were approved, and 43 of a total of 94 for public polytechnics.^ 

Due mainly to the extended self-regulating mechanism of the public universities, 
the provision of new study programmes suffers from a lack of national steering and 
effective coordination. The implication is a large and inflated higher education sector 
that was previously absorbed by an increasing demand, due to a high degree of student 
enrolment. However, this has now changed due to a considerable decline in the number 
of new students, and the situation calls for stricter regulation and a more consistent 
system of quality assurance. 

Combined with the development of recent years in Europe (in particular, the 
Bologna process), these factors formed the background for the Government decision to 
launch a new degree law in March 2006. An important element of this degree law is the 
provision of a legal framework for the establishment of a national accreditation system. 
The accreditation system is to guarantee a more efficient regulation of the provision 
of new study programmes and ensure a consistent system for the quality assurance of 
all existing study programmes within the higher education system. In chapter seven 
of this report, detailed recommendations for the design of a quality assurance system 
are provided. 
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Tertiary Education in Portugal; Background Report prepared to support the international assessment of the Portuguese system of 
tertiary education. A working Document, April 2006, MCTES (page 64). 


4- The existing quality assurance 
system for higher education 

4.1 Historical background 

Quality assurance of higher education in Portugal has been based on the Higher 
Education Evaluation Act (act 38/94), which was passed by the Parliament in November 
1994. The law was closely developed in cooperation with the Council of Rectors of 
Public Universities (CRUP) and forms the backbone of what is in Portugal widely 
termed “the contractual system”. 

As stated earlier in the report, the rapid development in the number of higher 
education institutions, particularly in the private sector, seemed to affect the quality 
of the education and raised the need for establishing a national quality assurance 
system. Especially the public universities regarded a national quality assessment 
system as a guarantee of their quality, vis-a-vis the private sector. In 1993, CRUP 
launched a pilot project involving five scientific disciplines: physics, computer sciences, 
electrical engineering, economics and French. The pilot project was designed with 
assistance from the Dutch agency for the evaluation of universities (VSNU) and the 
research institute CHEPS from the Dutch University of Twente. The pilot project 
was comprehensive and involved all public universities and the Catholic university. 
Simultaneously, a range of seminars were organised for the rectors, the faculty deans 
and the representatives of the institutions under review. The seminars were organised 
with assistance from VSNU and focussed on procedures for the preparation of self- 
evaluation and external evaluation. 

The procedures discussed were highly inspired by the so-called Dutch model. The 
Dutch university organisation VSNU had in 1988, in agreement with the Dutch 
government, set up an agency for cyclical evaluation of university study programmes. 
What the Dutch over the coming years presented across Europe as their model 
was partly based on the argument that a national external quality assurance agency 
must, like the VSNU, be owned by the universities themselves in order to ensure 
that evaluations furthered the improvement perspective of universities, and not the 
control perspective of government. This argument did not convince other contemporary 
European agencies, which were government owned, but still considered their activities 
to be improvement oriented. 

The Portuguese acceptance of the Dutch model was in terms of organisation 
reflected in the fact that a declared independent evaluation council was set up within 
the Foundation of the Portuguese Universities (FUP), resembling the model of VSNU, 
to activate the evaluation processes. The methodological framework, involving a two- 
phase process of self-evaluation followed by on-site peer review, was a copy of the Dutch 
model as well. 

The pilot project and the seminars were carried out in agreement with, and 
with support from the Ministry of Higher Education and provided the basis for a 
dialogue between CRUP and that Ministry. The latter was at that time responsible 
for higher education institutions. Later in 1994, this dialogue resulted in the approval 
and publication of the Evaluation of Higher Education Act. The act provided a legal 
framework outlining the general characteristics of the quality assurance system. 


Due to the complexity of the Portuguese higher education system (universities and 
polytechnic, both public and private), it was decided that the detailed requirements 
for a quality assurance system should be negotiated with each of the sub-sectors of 
higher education. In consequence, the Ministry signed protocols defining the rules of 
operation for quality assurance systems within each sub-sector, and these recognised 
the establishment of representative entities each with its own evaluation council: The 
Evaluation Council for Public Universities (FUP), The Evaluation Council for Public 
Polytechnics (ADISPOR); and The Evaluation Council for Private Universities and 
Polytechnics (APESP). 

The members of the evaluation councils include: a chairperson appointed by the 
representative entity; one person of recognised merit appointed by the Minister; three 
persons of recognised merit appointed by the representative entity; two professors 
appointed by the coordinating body of the respective higher education sector; 
one member of the representative entity’s governing body; and two students from 
the students’ unions. The fact that the membership was thus weighted towards 
representatives from the four higher education sectors did open the independence of 
the councils to questions. 

To co-ordinate these sub-systems, and to ensure the harmony, cohesion and 
credibility of the overall system, a meta-level organisation was established in the form 
of the National Evaluation Council for Higher Education (CNAVES). In that way, 
a complex and differentiated two-tier system for quality assurance was established. 
Already from the beginning, the system was associated with a number of challenges in 
terms of operational efficiency, consistency and independence. These challenges will be 
dealt with in more detail in the following sections of the report. 

4.2 Roles and responsibilities 

The two-tier model includes an operational level consisting of the evaluation council 
and a meta-level organisation with the overall responsibility for the coordination of 
the evaluation processes and for consistency between the sub-systems. The model is 
certainly complex, but the respective responsibilities of the two levels may be broken 
down as follows: 

The specific responsibilities of evaluations councils are to: 

• organise and coordinate the external review process; 

• propose to CNAVES the composition of the peer review panels; 

• propose to CNAVES the guidelines for the self-evaluation process; 

• monitor and support the development of the self-evaluation process; 

• ensure the publication of the external review reports as well as any refutations 
offered by the institutions whose study programmes are assessed; 

• issue suggestions and recommendations with the aim of stimulating quality and 
improving the evaluation model. 

The specific responsibilities of CNAVES are to: 

• appraise the global coherence of the quality assurance based on a comparative 
study of the evaluation reports; 

• appoint review panels based on the coordination of the proposals from the 
evaluation councils; 


• ensure that the same review procedures apply in each field of study, both in the 
public and the private sectors of each sub-system; 

• ensure that the same guidelines for self-evaluation are applied within each 
subsystem, allowing, however, for differentiation between the subsystems; 

• produce global and prospective reports concerning quality assurance in higher 
education as well as issue recommendations for the rationalisation and 
improvement of the higher education system. 

4.3 The content of the legal framework 

The Evaluation Act 1994 retains most characteristics of the Dutch model, primarily 
by placing the responsibility for the quality assessment system with an organisation 
representing the higher education institutions. In this way the evaluation system 
strengthens institutional autonomy, particularly that of the universities. In the process 
of importing the Dutch model, it was not taken into account that in the Dutch case 
a government inspectorate effectively supervised the processes and results of VSNU. 

The Ministry may have been expected to take such a role upon itself, but the Ministry 
obviously refrained from imposing on the CNAVES run system, even in cases where it 
had the formal background for doing so, as described below. 

The methodological framework is treated in more detail below in section 4.6. The 
self-evaluation of the programme under review is placed in the centre of the quality 
assurance process, followed by external review teams. This, too, is presented as a copy 
from the Dutch model. It should be mentioned, however, that this procedure was by 
rg94 already an element in the state-of-the-art practices in all European national quality 
assurance systems in place. Other characteristics similar to the Dutch model are that 
the system is national (all higher education institutions are involved), periodic (cyclical 
reviews) and comprehensive (all disciplines and study programmes will be assessed). 
Students are invited to participate in the quality assurance activities and the review 
teams are formally obliged to take into account the opinions of the students. The final 
evaluation report is made public and the draft report should be subject to a hearing 
process at the institutions responsible for the study programmes under assessment. 

The evaluations are cyclical. The first round (cycle) of external evaluations of study 
programmes took place between r995-2000 and included the public university sector. 
The polytechnics and the private universities were first included with the second round 
that took place between 2000-2005. 

The law clearly states that in the first round of evaluations there should be no 
direct link between the results of the evaluation and the level of funding. However, 
in case of successive negative evaluations or lack of follow-up on recommendations, 
the Government can take one or more actions in terms of decreasing or suspending 
funding, suspending degree awarding, etc. As is mentioned in section 4.10 concerning 
follow-up, the Government has never used this possibility. 

4.4 Organisational structure 

The two-tier organisational structure is complex and has created an organisation that 
suffers from a certain level of operational inefficiency and inconsistency. However, as 
stressed in several interviews during the site visit, the complexity in the system was 
borne out of the prevailing mood at the time it was established. Autonomy had very 
recently been won by the higher education institutions, and they had little inclination 


to reduce autonomy on a key issue such as external evaluations. Therefore strong 
interests among the higher education institutions argued to be represented at all levels 
of the system, while the government of the day avoided taking responsibility for the 
system. 

On the other hand it must be strongly emphasized that the contractual model, 
which reflects the large extent of institutional representation through the representative 
entities and evaluation councils, has created a strong sense of ownership and 
commitment to the quality assurance processes on the part of the institutions. This 
commitment would otherwise have been very difficult to achieve. 

Another justification for the complexity of the system relates to the institutional 
diversity of the Portuguese higher education system. The division into four evaluation 
councils has to some, even if certainly not full extent, contributed to a more equal 
representation between the four sectors (public/private, universities/polytechnics) in 
the quality assurance processes. During the site visit, this perspective was especially 
emphasised by representatives from the private sector and the polytechnics. They 
recognise this issue as one of the major advantages of the two-tier system. In other 
words, the divided structure has prevented a total domination by the public sector, 
especially the public universities (and CRUP) that are very influential in the Portuguese 
higher education system. 

The private institutions and the polytechnics have never obtained the same degree 
of political influence. For the latter, this is further underlined by the observation made 
during the site visit that the Council for Polytechnics, CCISP, is generally perceived as a 
significantly weaker political stakeholder than its university counterpart (CRUP). 

The major weaknesses in terms of operational inefficiency and inconsistency seem to 
be recognised by all relevant stakeholders including CNAVES. CNAVES itself critically 
analyses these problems in its self-evaluation report. 

The many levels of involvement in the evaluation processes cause the operational 
inefficiency. The various levels include CNAVES as the meta-organ, the evaluation 
councils as the operational organisations and the external experts (reviewers) that are 
further divided into a two-layer structure of review panels and review teams. The 
complexity in the operational structure is further underlined by the fact that MCTES 
has to approve the members of the review panels and teams. The many levels in the 
structure are time-consuming and provide a system of bureaucratic control, detracting 
from the quality improvement purpose. 

The limited staff capacity at CNAVES and the evaluation councils reinforce these 
weaknesses. As a consequence of the organisational structure, the technical and 
administrative support is fragmented into several secretariats. The few support staff 
(on average three) in CNAVES and in each council do not have the capacity to provide 
a proper quality control of the processes. The consequence is a limited degree of 
consistency in evaluation processes and in the final reports prepared by the various 
expert panels. 

Consequently, CNAVES has not been able to adequately fulfil its main role as a meta- 
organisation monitoring the global coherence of the quality assurance. 

4.5 Independence 

The independence of quality assurance agencies is a key requirement of the European 
standards. The standards emphasise that agencies must be independent from ministries 


as well as from higher education institutions and other stakeholders. Independence 
is, however, not a one-dimensional concept. This is partly due to the fact that 
in many European countries, including Portugal, higher education and the quality 
assurance agencies are funded and regulated by governments. Therefore, one important 
distinction lies between organisational independence on the one hand, and operational 
independence on the other. The standards do not imply that agencies should have no 
formal connections with ministries, higher education institutions or other stakeholders 
(i.e. be totally organisationally independent), but the operational independence of an 
agency should be unambiguous and guaranteed in official documents. 

Operational independence requires that the definition and operation of procedures 
and methods, the nomination and appointment of external experts, and the 
determination of the final outcomes, e.g. conclusions and recommendations, are 
undertaken autonomously and independently. 

The four elements that enable an understanding of operational independence can 
thus be summarised: 

• unambiguous guarantees in official documents; 

• independence in the definition and operation of methods and procedures; 

• independence in the nomination and appointment of external experts; 

• independence in the determination of outcomes. 

Guaranties in official documents 

The operational independence of CNAVES is partly assured through the legal 
framework regulating the quality assurance system. CNAVES was established with 
the Degree Act 1998 recognising CNAVES as the national coordinating body, with 
delegated responsibility for the credibility of a national quality assurance system. 
However, it can be claimed that the representative nature of CNAVES and the four 
councils has compromised the independence of the organisations from the higher 
education institutions. 

Definition and practice of methods and procedures 

CNAVES and the four respective councils have developed a common framework 
of guidelines for the self-assessment and external reviews. The guidelines are 
comprehensive and include all relevant aspects in relation to evaluation processes. The 
process by which the guidelines were developed is described in the self-evaluation 
report of CNAVES. Erom this it appears that the guidelines were developed and 
coordinated by a standing committee under CRUP with representatives from each 
university. The private universities were also invited to provide input through APESP. 
The evaluation councils for the polytechnic institutions (ADISPOR and APESP) were 
not directly involved in the formulation of the guidelines, but these two councils 
subsequently adopted the guidelines. Neither the Ministry nor the professional 
associations or other stakeholders were significantly involved in the discussion or 
preparation of the guidelines. 

The establishment and development of quality assurance systems has in most 
European countries included the necessary dialogue among authorities, agencies and 
stakeholders, including not least higher education institutions. If such a dialogue is 
based on mutual openness and trust it can be a strong mover towards a well working 
system. But it is also important that the system is credible in terms of its independence 


and therefore the line must carefully be drawn between dialogue with and undue 
influence from the higher education institutions. The involvement of representatives 
from the public universities as outlined above indicates clearly that the evaluation 
methods and procedures were not defined and developed in a manner that was 
sufficiently autonomous and independent from the higher education institutions. 

Nomination and appointment of external experts 

Independence in the nomination and appointment of experts is another critical issue 
in the Portuguese context. Formally, the appointment of experts is the responsibility 
of CNAVES. However, the appointment is based on recommendations from the four 
councils that invite the institutions (under assessment) to propose names. In the case 
of the public universities, proposals are submitted to FUP by the Advisory Thematic 
Committees in each of the study fields, which are composed of representatives from the 
member universities. Only in very few cases has CNAVES dismissed the nominations of 
the councils. MCTES finally approves the names. 

CNAVES has formulated a set of criteria for the selection of experts. One of these 
criteria concerns conflicts of interests and states that the experts may not have any 
formal relations with the institutions under evaluation or family ties with directors or 
teachers at the institutions. 

As indicated above, the formal procedures of CNAVES cover the issue of 
independence (no conflicts of interests) in the appointment of each specific expert 
team. However, the general process and the influence of the institutions in proposing 
names makes evident that appointment of experts does not take place independently of 
either the institutions or the Ministry. This seems to be generally recognised as a highly 
critical problem by most stakeholders engaged in quality assurance of higher education 
in Portugal, and an issue that was repeatedly confirmed as problematic during the site 
visit. 

It should be added, though, that the issue raised is not meant to imply that the 
experts involved necessarily lacked independence or honesty. The implication is that 
there are no formal procedures in the system to ensure that this is not the case. 

Determination of outcomes 

The final outcome in terms of conclusions and recommendations of the evaluation 
process should be the responsibility of the agency. However, this does not imply that the 
institutions under review are prevented from providing comments to the draft report. 
On the contrary, the higher educations should be consulted prior to the finalisation of 
the report. The procedure of CNAVES is in compliance with the European standard. 
The institutions are invited to comment on factual errors for correction or to contest 
the report by presenting a refutation. The review panel can then decide to introduce 
corrections to the report or to keep it unchanged. The final report and the refutations 
are published on the websites of both CNAVES and the respective council. The 
publication of the refutations adds positively to the transparency of the process. 

4.6 Methodological model 

The methodological model applied in the evaluation processes is thoroughly described 
in the self-evaluation report of CNAVES. Some of the elements that belong to the 


methodological model, e.g. nomination and appointment of experts, were addressed in 
the previous section and will thus only he presented here briefly. 

The elements in the model will be described and assessed against part two of 
the European standards and Guidelines (where relevant), which concerns external 
quality assurance of higher education. According to the guidelines (standard 2.4) 
the quality assurance agencies in Europe undertake different external processes for 
different purposes and in different ways. However, there are some widely used elements 
of external review processes, which not only facilitate the validity, reliability and 
usefulness of the agencies, but also form the basis for the European dimension to 
quality assurance. These elements concern: 

• Institutional improvement as a fundamental element in the quality assurance; 

• Experts; 

• Student participation; 

• Self-evaluation/site visit/draft report/published report/follow-up as the significant 
elements in external evaluation processes. 

Institutional improvement 

According to the CNAVES self-evaluation report, the main objective of the quality 
assurance system is to contribute to the improvement of quality within the higher 
education institutions, while fostering responsibility and accountability towards society. 
The element of self-evaluation validates the improvement perspective in the evaluation 
process. The site visit provided clear evidence that the self-evaluation processes 
have brought a cultural change to many higher education institutions and a greater 
awareness of democratic processes, e.g. how to include students in the development 
of the institutions. The evaluations, especially those conducted in the first round 
of evaluations from 1995-2000, have contributed to building up a self-evaluation 
culture at many institutions, which has continued after the completion of the external 
evaluation process. 

The improvement perspective was seemingly stronger in the first round. That phase 
was considered as an experimental phase focusing on the learning process to build up 
ownership towards an external quality assurance system, and to develop an evaluation 
culture at the institutions. Institutional improvement was a major concern at this point, 
while the issue of accountability played a secondary role. 

However, this changed gradually with the second round which took place from 2000 
to 2005. In that round, a rating scheme to reinforce the accountability dimension 
was applied on a trial basis. The study programmes were assessed against fourteen 
fields and achieved an assessment score within each held ranging from “excellent” to 
“negative”. The rating system, however, never worked as intended due to a high degree 
of inconsistency in the reports and in the interpretation of the ratings by the various 
expert panels. The issue of inconsistency will be further elaborated in the following 
section on experts. 

Profile of experts 

The issue of independence in the nomination and appointment of experts is addressed 
in section 4.5 of the report and will thus not be discussed further here. 

The reliability of an external evaluation process depends to a high degree on the 
competences and credibility of the expert teams. According to the European standards. 


attention should be given to the skills and competences of the expert teams, to the 
careful selection of the experts, to the provision of appropriate briefings and training, 
and to the use of international experts. 

According to the self-evaluation report, CNAVES gives priority to experts that are 
recognised as experts in their field of research and who possess a certain academic rank 
(preferably full professors). On the one hand, these concerns are justified in order to 
achieve the necessary legitimacy, particularly at the universities. On the other hand, 
they favour one specific type of competence at the expense of other highly relevant 
professional skills such as experience with quality assurance and knowledge of the 
labour market. Moreover, CNAVES and the four councils do not have mechanisms to 
verify the assumed quality of the skills and competences of the experts by inviting 
appraisals of potential experts from other independent sources. The performance of the 
experts is evaluated by the chairman of the panel, but the performance of the chairman 
is not itself subject to evaluation. 

The CNAVES guidelines for external reviews of both subsystems address explicitly 
the organisation of the work of the review panels, including the meetings to be held 
by the expert team. The type and frequency of the meetings seem appropriate and 
sufficient, and include: 

• a preliminary meeting within ten days of receipt of the self-evaluation report to 
discuss the preparation of the external review, including working methodology, 
division of tasks, analysis of the self-evaluation and identification of the need for 
further information from the institutions; 

• a second meeting that takes places three weeks after the preliminary meeting, 
with the aim of deepening the analysis of the self-evaluation reports and preparing 
comments and questions to be addressed during the site visit; 

• a third and final meeting to discuss and approve the report from the external 
review. 

While the frequency in meetings between the panel members is appropriate and 
comparable to standards of other European agencies, the training of experts is 
insufficient, with implications for the consistency of the reviews. Due to the limited 
staff capacity of CNAVES and the four councils, training is limited to one annual 
preparatory meeting. Eor the university sector, this meeting includes, in principle, 
all members of the panel, whereas in the polytechnic system, it only includes the 
chairpersons and vice-chairpersons. However, during the site visit, the ENQA panel 
gained the impression that it was only in a few exceptional cases that all members 
(or a majority of them) attended this meeting. In a quality assurance model like the 
Portuguese one, where the responsibility for drafting the external reports relies on 
the external reviewers, proper training and briefing is crucial in order to obtain the 
required consistency and level of professionalism. 

In this respect, CNAVES’s evaluation process has failed. One example to illustrate 
the implication of insufficient training is the differentiation in the application of the 
rating scheme that was introduced in the second round of evaluations. According to the 
CNAVES self-evaluation report, the rating scheme has led to somewhat heterogeneous 
approaches by the different review panels. The inconsistency is also seen in the 
structure of the reports, where great varieties pose difficulties in the comparison of 
results. There is an obvious risk that the experts are subjective and biased in their 


approach, with a tendency to view and evaluate a programme on the basis of their own 
home institution programme. 

One frequently noted weakness of the experts is the degree of reliance on reviewers 
from Portugal. CNAVES stresses that a measurable percentage of international 
evaluators was utilized in the reviews and increased over time. CNAVES also notes that 
it did not have the authority to require reviews in English, which would be necessary 
for foreign teams. Many of those interviewed observed that Portugal was a “small 
country”, and that reviewers and the reviewed frequently knew each other well, and 
this sometimes compromised the independence and objectivity of the external reviews. 
Also that many Portuguese faculties are in-bred (composed primarily of their own 
graduates), and, therefore, that Portuguese higher education needs the stimulation 
provided by the introduction of European standards and practice. 

Private institutions were particularly outspoken in the need for international 
reviewers. They noted that almost all of their external reviewers came from the 
Portuguese public sector (i.e., their competitors) and often appeared biased against 
private programmes. Einally, while there was strong support for international reviewers 
among both private and public institutions, there was also widespread support for 
CNAVES’ view that members of external review teams need to be knowledgeable about 
Portuguese culture and the system of higher education. 

An increased use of international experts would of course add substantially to the 
costs and work involved in the process, but, at the same time, would create a much 
more credible quality assurance system for two main reasons. Eirstly, it would add 
an international comparative dimension to the external quality processes, which is 
normally not achieved by using purely national expert teams. Secondly, it would fortify 
the impartiality and independence of the review teams by expanding the potential pool 
of experts. 

It should be added that for a number a years a very strong argument has been 
established for adopting international experts, as these have been the backbone of the 
external reviews initiated by the Portuguese government for research units. The ENQA 
panel had during the site visit the opportunity to meet several key individuals involved 
in, or with experience of, the research review exercise. The feedback to the panel was 
very clearly that the exercise had been successful and credible especially due to the use 
of non-Portuguese experts. Obviously the research reviews have acted very positively as 
catalysts for an increased awareness of the implications of successful external reviews. 

Student participation 

In Portugal, student unions are an important actor in the higher education system, as 
a consequence of the 1974 revolution, hollowing this event, students now participate at 
all levels of institutional governance. 

In the quality assurance of higher education, students are also formally playing an 
active role at all levels. Students are formally represented in CNAVES, the four councils 
and other committees engaged in quality assurance, e.g. the Bologna Committee, but 
their real influence is limited. During the site visit the students argued that they are 
not being listened to in CNAVES or the councils, where the majority of the members 
are senior academics. 

In the quality assurance processes, the students have never managed to manifest 
themselves as a key player - neither in the internal self-evaluation processes nor 


as members of the external review teams. However, the role of students in quality 
assurance is not only constrained by the resistance of the institutions to involve 
students, but also by the fragmented structure of the students’ organisations and by a 
degree of non-assertiveness and lack of commitment on the part of the students. A less 
differentiated organisation would potentially provide a stronger basis for the influence 
of students at all levels. At the same time it must be added that the influence of the 
students in the quality assurance process constitutes also a problem in a large number 
of European national quality assurance systems. 

Self-evaluation/site visit 

Appropriate and comprehensive guidelines for the self-evaluation and site visit have 
been prepared by the evaluation councils for both subsystems and approved by 
CNAVES. 

The guidelines for self-evaluation are almost identical for the two subsystems but are 
slightly more detailed for the polytechnics, the latter perhaps due to the fact that the 
polytechnics first joined the existing quality assurance system in the second round from 
2000 to 2005. The guidelines reflect a proper balance between the need for factual 
description and critical self-evaluation. Moreover, the guidelines include a standardised 
format for the presentation of data in order to facilitate a systematic and homogeneous 
presentation of data. In addition to the format, the guidelines include procedural 
advice on, for instance, the composition of the self-evaluation groups. Concerning 
the guidelines to institutions, CNAVES has confined itself to recommendations and 
has acknowledged that the composition of the self-evaluation groups is an internal 
matter for the institutions. The ENQA panel does, however, consider it a mistake that 
institutions did not receive clearer directions on these and other issues. One obvious 
example could be to have unmistakably conveyed to institutions that students should 
join the self-evaluation groups and be given an active role. 

The procedures for site visits are defined in the guidelines for external reviews and 
include all relevant aspects in relation to the organisation of a site visit. The procedures 
include a format for the terms of reference for the site visit, interview protocols and a 
draft site visit schedule. 

In summary, the guidelines for self-evaluation and site visit are generally appropriate 
and in compliance with the European standards. 

Report 

The external review reports are drafted by the various expert teams themselves, and the 
staffs of the evaluation councils have no role in the drafting or approval of the reports. 
Guidelines for the structure of the reports exist, but they are often not followed by the 
various teams, as was made evident during the site visit. The result is an obvious risk 
of inconsistency in structure as well as style. In other words, the variety of the reports 
makes it difficult for general readers to combine and compare results of the individual 
reports and to arrive at an understanding of quality across programmes. 

The writing style of the external review reports is another concern, which was raised 
by a number of persons during the site visit. The reports are often formulated in a vague 
manner, without clear conclusions and recommendations. This can either be explained 
by the limited impartiality of the review teams (friends evaluating friends) or as a result 


of an inconsequential evaluation system. This latter issue will be further elaborated in 
section 4.10 regarding follow-up. 

Accordingly, the format and content of the reporting does not fully comply with the 
European standard 2.5 which stresses that the reports should be written in a style which 
is clear and readily accessible to its intended readership. The standard further stresses 
that any decisions, commendations or recommendations contained in reports should be 
easy for a reader to find. 

Publication of final report 

The final reports are published on the web sites of both CNAVES and the evaluation 
councils in compliance with the European standards. 

CNAVES has, however, seemingly not been very successful in making evaluation 
processes and results apparent and known to the Portuguese society. 

4.7 Focus 

The guidelines for the self-evaluation and external reviews include relevant concerns 
about programme curricula, teaching and student performance as well as quality 
assurance. However, these guidelines are rather different from the Dutch approach that 
served as the model for the reviews, and which was particularly concerned with the 
quality of teaching and student learning in academic programmes. The categories of 
the CNAVES reviews, as well as the sample reports themselves, focus much more on 
programme capacity, including the numbers and compositions of academic faculties 
and staffs as well as the adequacy of facilities and equipment. The Dutch reviews 
focused to a greater extent on programme goals, the structure and content of the 
curriculum, student assessment, the quality of student work, and methods of quality 
assurance. 

This difference is also obvious in the global or summary reports produced in the two 
different countries. The Dutch published reports that included an extensive discussion 
of the nature of teaching and curricula in each subject field, while the CNAVES 
global reports appear to primarily contain summaries of the team judgments of each 
programme. The potential for using the study programme reviews to share issues and 
discovered best practices in curricula content, teaching, and student assessment in each 
subject field appears not to have been exploited. In comparison, the teams carrying out 
the Portuguese research evaluation reports included discussions of the state of research 
in each field assessed. 

Predictably a number of Portuguese institutional leaders interviewed during the site 
visit reported frustration with the orientation of the CNAVES review recommendations, 
which seemed to emphasize personnel, resource, and facilities issues that they as 
institutional leaders could do little to address, rather than the academic curricula, 
teaching, and assessment issues where self-improvement might have been more feasible. 

4.8 Coverage and scope 

In the first round, a total of 376 study programmes were assessed, involving 14 public 
universities, the Catholic University and one university institute. 


In the second round the number of study programmes assessed increased to a total of 
1,209 divided between 596 in the university sector (432 public and 164 private) and 613 
in the polytechnic sector (421 public and 192 private)®. 

The first evaluation round between 1995-2000 was narrowly focussed on the 
programme level. From the second round, the focus gradually shifted to a study-field 
approach with grouping of study programmes (but still with a programme-oriented 
approach). For the university programmes, 44 fields of study were identified and 
the polytechnic system was grouped in 12 wider areas. A third round was planned 
from 2005, but in light of the Government’s initiative to undertake a comprehensive 
international review of the higher education system and the quality assurance system, 
the process was put on hold. The panel received many comments on this during 
the site visit, and not only from CNAVES. The general point made was that the de 
facto moratorium on external reviews was highly problematic and de-motivated the 
champions of CNAVES’s evaluation process - not least because CNAVES was pre- 
empted from putting its own revisions of its system in place. 

The cyclical basis on which the study programmes were evaluated complies with 
European standard 2.7 concerning periodic reviews. 

The level of coverage increased from the first to the second round. 376 university 
study programmes were assessed in the first round, and 596 in the second. This could 
indicate that the approach of grouping increased the efficiency of the system and thus 
the level of coverage, but the explanation is rather that the private university sector 
joined the evaluation system for the second round. 

The panel is puzzled by the information received that there were no consequences 
in cases where higher education institutions simply did not wish to submit their study 
programmes to evaluation. Unlike the Ministry CNAVES had no tools to enforce 
participation. However it informed the Ministry regularly. 

4.9 Involvement of stakeholders 

Employers’ organisations and representatives have not been very visible in the current 
quality assurance system. Even if employers have had some representation in CNAVES 
or in the evaluation councils their activity and participation have been low. Employers 
have been invited to join panels, though not to very great extent and recruitment has 
been difficult because of the low priority given by many employers to this kind of 
work. In the polytechnics sector, however, institutions have included questionnaires to 
employers as part of the self evaluation process. 

4.10 Follow-up 

A widely acknowledged weakness of CNAVES’s evaluation process is the lack of 
consequences of or follow-up on the reviews. CNAVES makes this a central point in its 
self evaluation report and argues that this was a failure on the part of the Ministry. 

In The Netherlands, for example, which was a model for the Portuguese process, the 
reviews made by the university agency, VSNU, were followed by a site visit by the 
Inspectorate for Higher Education, which first checked to see whether the university 
had developed an “Action Plan” in response to the evaluations, and checked back again 
several years later to ensure that the quality improvement plans were implemented. 


The self-evaluation report (page 31). 


Follow-up visits to discuss reviews and to explore what institutional actions had been 
taken in response to reviews might also have been undertaken by CNAVES itself and 
would have been consistent with the professed goal of academic self-improvement. 
However, in fairness, effective follow-up to quality reviews has been an issue in many 
EU countries and until recently most countries in practical terms, or even formally, 
delegated responsibility for the implementation of review recommendations to the 
institutions themselves. 

The cyclical basis of the evaluations should in principle provide some sort of feed- 
back on the consequences from the previous evaluations, but this was evidently not the 
case and there seems to have been a complaint from the public universities that reviews 
in the second round did not cover or reflect results from the first round. 

In summary, the current system for follow-up is not in compliance with European 
standard 2.6 concerning follow-up. 

4.11 Information on labour market and employability 

Valid Information on employability and labour market exists in Portugal and is 
collected by the Ministry for Employment and Professional Training on a regular basis. 
The data collected is, however, not published or distributed to the higher education 
institutions or coordinated with the quality assurance processes of CNAVES. The panel 
finds this to be a serious shortcoming of the current system. The collection and use 
of data relating to the labour market would strengthen the capacity of institutions to 
improve their academic study programmes. Just as important is the use of output data 
in the external quality processes to verify the assessment of the quality of the study 
programmes. 

4.12 Major conclusions on strengths and weaknesses 

The ENQA panel has in the preceding pages identified the essential strengths 
and weaknesses of CNAVES. Many of these, not least the weaknesses, were, very 
commendably, already identified in the CNAVES self-evaluation and are thus validated 
by the ENQA review. 

Major strengths and considerations for carrying-over 

The evaluation also identifies a number of strengths in the quality assurance system 
that the Portuguese Government should consider carrying-over into a new quality 
assurance system. These strengths are summarised in the following. 

• CNAVES’s evaluation process has introduced into Portuguese higher education 
the concept and process of self-evaluation. In a number of institutions, surveys 
among students had not been conducted until required by the CNAVES reviews. 

A number of institutions have developed their own internal quality assurance 
systems as a result of the CNAVES reviews. The CNAVES reviews further 
assisted many institutions in becoming more reflective about their practice and 
the institutions have made their education programmes more transparent to 
the public and to students. This was particularly true in the private sector, 
where institutions were generally quite new and had little experience with self- 
evaluation. 


• The ENQA panel therefore believes that an emphasis on institutional self- 
improvement as well as quality enhancement should also be an important part 
of any future quality assurance system for Portugal. This consideration is further 
elaborated in the next chapter concerning recommendations for a future system, 
where it is suggested that this is realised through a comprehensive quality 
assurance system covering programme accreditation and institutional academic 
audit. 

• CNAVES’s evaluation process (self-evaluations of subject fields, external reviews 
by teams of experts, and published reports) is consistent with accepted 
international quality assurance practice and specifically with the European 
standards. The methodological model should, therefore, in outline continue in a 
future quality assurance system. However, the future model should ensure the 
involvement of students and other stakeholders to a greater extent; until now, 
these have been almost absent in the quality assurance processes. This concern is 
included in the recommendations in the chapter seven. 

• Private institutions indicated that their inclusion in the CNAVES reviews 
helped improve the recognition and reputation of their sector. To some extent, 
the division into four evaluation councils has contributed to a more visible 
representation in the quality assurance processes between the corresponding four 
sub-sectors. 

The major weaknesses identified by the panel include: 

• The independence of CNAVES operations in relation to the reviewed institutions 
was perceived as a key critical issue in most of the ENQA panel’s interviews 
outside the CNAVES constituency. The essential ownership of CNAVES by 

the higher education institutions and the low level of government attention 
contributed to the image of the CNAVES reviews being too uncritical towards 
the higher education institutions, thus bringing into question the value added, in 
terms of quality assurance, of the many reviews. 

• Variability and inconsistency in the focus, conduct and reporting of the external 
teams was perceived to be a common problem, although there is some evidence 
that the consistency and reliability of the CNAVES external reviews have 
improved over the two cycles of implementation. The limitations on training 
was probably a factor, but the highly decentralized structure of CNAVES and 

its dependence upon collective meetings as a means of coordination proved to 
be quite expensive and also likely to have contributed to the inconsistencies 
among the reviews. This decentralized organization also made it more difficult 
for CNAVES to develop and utilize a professional staff in quality assurance. A 
staff with such qualifications is a common characteristic of effective agencies in 
other EU countries. 

• Another frequently noted weakness of the external reviews was the heavy reliance 
on national Portuguese reviewers. The main problem in this regard is that 
Portugal is a “small country”, and that reviewers and the reviewed frequently 
knew each other well and this sometimes compromised the independence 

and objectivity of the external reviews; many Portuguese faculties are in-bred 
(composed primarily of their own graduates) and, therefore, Portuguese higher 
education needs the stimulation provided by the introduction of European 


standards and practices. Moreover, CNAVES and the four councils do not have 
mechanisms to verify the assumed quality of the skills and competences of the 
experts by inviting appraisals of possible experts from other independent sources. 
A fourth crucial deficiency of CNAVES’s evaluation process is the lack of 
consequences, or follow-up on the reviews. The passiveness on the part of the 
Government is part of the explanation, but CNAVES’s lack of activity does also 
bear part of the responsibility. Eollow-up visits to discuss reviews, and to explore 
what institutional actions have been taken in response to reviews, could also have 
been undertaken by CNAVES itself, and this would have been consistent with the 
professed goal of academic self-improvement. 

Einally, CNAVES’s evaluation process, publications, and meetings provided 
less evidence than expected of what is described in other countries as 
“quality enhancement”, i.e., encouraging the identification, communication, and 
implementation of “best practices” in curricula design, teaching, student learning, 
and institutional academic quality assurance. 


5- Accreditations conducted by 
professional associations 

At a time, when there is no national systematic accreditation system in place, neither 
run by the state nor by an independent agency, a number of professional associations 
conduct or plan to conduct professional accreditations. Like the CNAVES reviews 
the professional accreditations only deal with the programme level. However, the 
professional accreditations are in no way coordinated with the CNAVES reviews. 

The interest of the professional associations in accreditation is primarily due to the 
recent and projected turndown in student enrolments that has enhanced competition 
across the higher education sector and created an apparent mismatch between the 
outputs of the higher education and the needs of the labour market. Combined with the 
perceived deficiencies in the current national quality assurance system, concerning lack 
of consequences or follow-up on the CNAVES reviews, these developments have further 
influenced the interest among the professional associations in the regulation or control 
of market access for their candidates. 

Below, the scope, methodological model, consequences and main strengths and 
weaknesses of the professional accreditations are outlined, as well as the views of the 
professional associations on a future national accreditation system. 

5.1 Professional associations conducting accreditations 

There is a large number of professional associations in Portugal, including: pharmacists, 
economists, engineers, lawyers, nurses, doctors, veterinary doctors, dentists, architects, 
biologists and certified accountants. Only three of these are currently pursuing 
professional accreditation: the associations of engineers (Order of Engineers), architects 
and pharmacists. The documentation reviewed by the ENQA panel clearly shows that 
the level and extent of accreditation activities vary significantly among these three 
associations and that their accreditation procedures are at very different stages of 
development. 

The association of engineers (Order of Engineers) is obviously the more significant 
player and has certainly the most well developed accreditation procedure. The 
association was also the first among the professional associations to establish an 
accreditation procedure in 1994. There is a parallel association for technical engineers 
called ANET that, until recently, offered accreditation of three-year technical engineer 
courses offered by the polytechnic institutions. However, ANET’s accreditation 
activities are for the time being suspended. The association of architects established 
an accreditation system in 1999 similar to that of the Order of Engineers, while the 
association of pharmacists established their accreditation procedure a few years later 
in 2001. 

No other professional associations have so far established a system of accreditation, 
and the articles of association of some do not allow them to. However, several 
associations are in the process of changing their articles in such a way that they 
will be allowed to pursue accreditation. This applies, for example, to the association 
of veterinary doctors. The association already has an EU-wide accreditation process 
in place and has adopted European minimum standards for veterinary surgeons. The 


panel has noted that the veterinary association is the only one in Portugal with a clear 
European focus that will automatically involve a process with EU-wide standards as 
well as an international external review team. 

5.2 Methodological model 

The professional accreditations rely on a methodological model that, at a general level, 
is similar to the one of the existing quality assurance system and the model of most 
European quality assurance systems: accreditation is cyclical and is supported hy a 
self-evaluation report and a site visit hy an external review team. 

However, one distinct difference is the non-puhlication of the final report. While 
the CNAVES reports are public, the accreditation reports are held confidential and 
can only be made public with permission from the institution being accredited. In 
this regard, the professional accreditation in Portugal is not in accordance with the 
European standards and other European accreditation procedures. 

The accreditation process is lengthy and takes on average 13 months. 

A professional accreditation has usually three possible results: 

• accreditation for a period of six years, when renewal of the accreditation will be 
necessary; 

• conditional accreditation for a period of three years with recommendations for 
improvements that need to be implemented within the period and confirmed by 
a review team; 

• refusal of accreditation (non-accreditation). 

The three forms follow the pattern of most other European accreditation systems and 
strengthen the credibility of the system. The reason for this is that in a system with only 
a yes/no option, the accrediting body would often refrain from refusing accreditations 
because of the serious implications for the institutions concerned. The vast majority 
of the programmes would thus receive an accreditation, and this would significantly 
compromise the credibility of the system. 

In addition, the inclusion of conditional accreditation adds an improvement 
dimension to accreditation that is basically more control-oriented than other external 
quality assurance methods, such as evaluation and audit. 

5.3 The consequences of professional accreditations 

The professional accreditations are not formally linked to funding or degree awarding 
due to the voluntary nature of the system. However, the accreditations do have clear 
consequences. 

Refusal of accreditation implies a loss of prestige and of the capacity to enrol students 
in the study programme, with a lower budget as a consequence. 

Awarding of accreditation of a programme has the important result that it is 
made easier for graduates of the programme to become members of their respective 
associations. This is of high importance to the graduates, as membership of the 
association is often a prerequisite for exercising their profession. 

The requirements for becoming a member of a professional association vary 
substantially from association to association. In some cases, holding the appropriate 
degree is a sufficient condition, while in others admission is first allowed after a 
training period and/or and examination. The latter situation is where accreditation 


comes to mean a great difference for the students, as graduating from an accredited 
programme in some professions (e.g. engineering) allows the students to skip this 
training period and/or examination and automatically become a member of the 
association. 

The implication is that students graduating from accredited programmes often have 
a competitive advantage compared to students from programmes that have either 
not completed an accreditation process or have received a conditional or refused 
accreditation. The professional accreditations thus create rather unequal conditions for 
the students - a fact that especially the student representatives, with whom the ENQA 
panel met during the site visit, were very concerned about. 

5.4 Strengths and weaknesses 

The ENQA review panel has identified three particular advantages of professional 
accreditation in comparison with the CNAVES reviews. The analysis is especially based 
on the example of the engineers association 

Eirst of all, the accreditations have consequences and a high degree of credibility, 
as demonstrated by the results from the more than 200 accreditations of study 
programmes that the Engineers’ Association has carried out in the period 1994-2005. 
Of these 33 achieved a negative accreditation. 

Secondly, it is an advantage of the system that the level of credibility is closely linked 
to the consistency between processes. Consistencies in the processes and reports are 
created through the involvement of staff from the professional association in the site 
visit and by careful reviewing of the reports by the professional staff of the associations. 
The engineers association plans to strengthen the consistency element further by 
handing over the responsibility for drafting the reports to the staff of the association. 
Report writing is until now usually the responsibility of the external review teams. 

Thirdly, the criteria for the professional accreditations are focused on student 
learning outcomes as well as educational input. This outcome focus is in contrast to 
the CNAVES reviews that mainly emphasise input indicators like personnel, resources 
and facilities. 

The professional accreditations do have some evident weaknesses as well: 

Eirst of all, the professional associations rely heavily on national experts. This 
was mentioned several times during the site visit as a deficiency of the professional 
accreditations. However, with increasing international cooperation and the adaptation 
of international criteria and standards, more international experts are likely to be 
included in future expert panels. Eor instance, the engineers association is involved in 
the EUR-ACE cooperation and has recently revised the standards of the association to 
ensure compliance with those of EUR-ACE. Also, as mentioned earlier, the association 
of veterinary doctors has an EU-wide accreditation process already in place that will 
involve international review teams. 

Secondly, the coverage of the accreditations is quite low. The engineers association 
reported coverage of only 40% after 11 years of accreditation. The Architects’ 
Association has completed 25 accreditations from 1999-2005 and the Pharmacists’ 
Association has conducted seven from 2001-2005. 

Thirdly, the transparency of the accreditation processes is limited due to the 
previously mentioned problem that the reports are not made public. This makes 


the accreditations out of step with the development in European quality assurance 
practices. 

5.5 Future 

The panel received many comments during the site visit on the obvious lack of 
coordination of the processes of CNAVES and the professional accreditations. In 
particular, a number of institutional representatives expressed their frustration with the 
lack of coordination and pointed to the double impact and overloading of institutions 
that were responding to the requirements of both processes. Another noted frustration 
was related to the disturbing fact that the two processes in many cases lead to different 
conclusions on the quality of individual programmes. Some professional associations 
shared this frustration and expressed the need for one national mandatory accreditation 
system. They agreed that the capacity to pursue accreditation should be vested in 
the new national agency for accreditation, and that the accreditation power of the 
professional association as a consequence should be transferred to the new agency. The 
risk of having two systems running in parallel was too evident. 

However, the associations also stressed that they should continue to play a role in 
accreditation and that the valuable experience of many years of accreditation deserves 
early consideration by the new agency. In the chapter on recommendations for a future 
Portuguese quality assurance system, specific recommendations for the role of the 
professional associations in the future system are provided. 


6. Conditions for a new Portuguese 
quality assurance system 

6.1 Mandate of the ENQA panel 

ENQA has been asked by the Portuguese government to provide recommendations 
that will lead to the establishment of a national accreditation system and practices 
which will meet the European Standards and Guidelines for Quality Assurance in 
the European Higher Education Area (EHEA) and ensure appropriate academic and 
management structures for implementing adequate quality assurance and accreditation 
practices. 

This initiative from the Portuguese government parallels activities in a growing 
number of European governments that are applying quality assurance systems for 
higher education that are based on an accreditation approach. At the same time, it is 
notable that there are often variations in the interpretation of what the exact resulting 
demands and implications are in terms of organisation and basic methodological 
choices. 

It is an important recent development that European external quality assurance 
agencies have, to an increasing extent, accepted the common denominator “quality 
assurance” as the cover term for methodological specialisations such as evaluation 
or accreditation. The decisive expression of this development is the consistent use 
of the term “quality assurance” in the ENQA report to the ministerial meeting in 
Bergen 2005, rather than combinations such as “quality assurance and accreditation” 
or “accreditation and evaluation”. The ENQA panel considers the consequence of this 
development in section 7.1. below on quality assurance terminology. 

6.2 The basis for the recommendations of the ENQA panel 

In fulfilling its task the ENQA panel has drawn on its members’ considerable experience 
of international and European best practices within quality assurance, including 
accreditation. Eurthermore, the panel has based its considerations on its high degree 
of familiarity with the content and intentions of the European standards of quality 
assurance adopted at the Bologna conference of ministers in Bergen in May 2005, at 
which three panel members were very involved in the formulation and agreement of 
the standards. 

Accordingly, it is a basic premise for the ENQA panel that the recommended future 
system must meet the code of practice expressed in the European standards. 

Therefore the recommendations of the panel are based on the view that the coming 
Portuguese accreditation system of higher education should not deviate from the 
fundamental principles on which quality assurance of the higher education system is 
now based in most European countries. 

The following are basic elements of this practice. 

• The process of quality assurance is independent of public authorities as well as 
education institutions and is, therefore, normally based in an independent quality 
assurance agency. 

• External experts carry out the academic assessment. 


• As a main rule, the basis of the assessment will be a self-evaluation report 
prepared by the programme under review in combination with a site visit. 

• The process is completed by the publication of a report. 

The ENQA panel has further carefully considered strengths and weaknesses of the 
existing system as reflected in the first section of this report. The strengths are identical 
with those good practices and experiences that will benefit the new accreditation 
agency. The weaknesses point to those procedures and elements of organisation that 
provide valuable learning experiences, but that should not be carried on into the new 
system. The analysis and recommendations in this section will therefore also be based 
on the links to the CNAVES experience as reviewed by this panel. 

The ENQA panel made use of its site visit week to discuss with a broad range 
of stakeholders their views on and priorities for a future quality assurance system 
for higher education. The panel must make its own analysis and arrive at its own 
conclusions, but it was struck by the overall consistency of views and expectations 
expressed. The main points may be summed up in the following statements. 

• There is a distinct need for an accreditation system to adequately place Portuguese 
external quality assurance of higher education in an international or European 
framework. 

• The European standards for quality assurance of higher education should form the 
basis of the future system. 

• The new system must be constructed in a manner that ensures credibility 
based on the necessary independence of operations from the higher education 
institutions as well as from government. 

• The new system should cover the programme level, but be paralleled by a system 
of academic audit and accreditation at the institutional level. 

• The improvement perspective should be maintained in the new system. 

• The processes should, as a rule, involve international experts. 

• The new agency must ensure consistency of its processes and therefore be strong 
in terms of professional administration and staff. 

• The new agency must respect the structure representing public/private 
universities/polytechnics. 

• The new agency must cover all study programmes of higher education, and, 
accordingly, the accreditations carried out by the Portuguese professional 
associations must cease as independent activities. 

• The new agency must be consistent and possess well-defined and operational 
procedures for follow up on. 


7- Key issues and recommendations 
for a Portuguese quality assurance 
system 

7.1 Quality assurance terminology 

In the following sections, the ENQA panel will analyse and discuss a number of key 
issues that are relevant for the coming establishment of a Portuguese accreditation 
system, and present its resulting recommendations to the Portuguese government. 
However, the broader heading of quality assurance is applied by the panel to 
present its analysis, because this leads to conclusions that point towards the need 
for a comprehensive Portuguese quality assurance system that includes not only 
accreditation, but also academic audit. An academic audit is an external review of the 
internal quality assurance processes and mechanisms that academic institutions employ 
to assure the academic standards of their study programmes. 

The ENQA panel recommends that the name of the new agency reflects the 
recommendation below that the agency covers accreditation and evaluation/audit. The 
panel’s specific recommendation bears the title “The Quality Assurance Agency” and not 
“The Accreditation Agency”. 

The quality assurance name is compatible with the terminology used in the European 
Bologna process (see the ENQA report adopted at the Bergen meeting of Ministers 
2005). At the same time, the name opens up the possibility for further methodological 
diversification of the agency. Innovation must be possible in order that the new 
Portuguese agency can adopt other methodological approaches than accreditation, for 
example audit. 

Internationally, accreditation is often defined and interpreted differently. In several 
national systems accreditation processes are equal to what in Europe is called 
evaluations, i.e. quality assurance processes that are not based on criteria and do not 
lead to authoritative decisions on quality levels. The ENQA panel, however, finds that 
accreditation differs from evaluations primarily in terms of its point of departure and 
outcome. 

The ENQA panel recommends, therefore, that the new agency adopts the following 
definition of accreditation: The characteristic of accreditation is that the process is based 
on established external criteria and that the result of the process is a decision as to whether 
these criteria have in fact been met by the institution or programme under accreditation. 

Whether labelled accreditation or something different, the important point is that the 
method used is based on agreed and published criteria, and makes a formal statement 
about whether or not the evaluated object meets these criteria. 

Further accreditation is used with two different meanings: signifying either a stamp 
of approval/disapproval or describing a method and process of quality assurance. To 
indicate that an institution or programme has been awarded the stamp of approval, the 


term ’’accredited” is used, and, likewise, the term ”non-accredited” is applied in the 
event that an institution or programme fails to achieve such a stamp. 

A distinction is made between pre-accreditation (ex-ante) and accreditation (ex-post). 
Pre-accreditation is a preliminary review of the basis on which a programme is to be 
established and will result in either a yes or no to launch the programme. Such reviews 
may be based on desk research reviewing study regulations, the availability of qualified 
teachers, admission requirements and facilities. Pre-accreditation usually replaces the 
ministerial approval of new study programmes. 

The ENQA panel recommends further that academic audit be defined by the new agency as 
a method for evaluating the strengths and weaknesses of the quality assurance frameworks 
and mechanisms adopted by a higher education institution for its own use in order to 
continuously monitor and improve its activities and results. 

The fundamental issue in academic auditing is how an institution knows that the 
standards and objectives it has set for itself are being met. It should also be stressed that 
academic audit has accordingly a very strong enhancement dimension. 

7.2 National versus international accreditation 

The purpose of accreditation in the new quality assurance system is partly the obvious 
one of assuring that quality at higher education institutions meets the requirement 
of the government and stakeholders, and provides institutions with benchmarks on 
which to base their further development. But accreditation must also make the 
quality of Portuguese higher education programmes transparent and comparable in an 
international context. Only in this way, can Portuguese higher education secure its 
place in the global context of higher education and, thereby, the necessary international 
mobility for staff and students. 

Higher education has namely in a global context begun to take on some of the 
characteristics of a market place, and the new accreditation system must be designed 
to support the efforts to attract the best staff and students from abroad, and to provide 
access for Portuguese staff and students to academically attractive institutions abroad. 

In this context, considerations of the new Portuguese system must also take 
into account that external quality assurance of higher education is increasingly a 
global affair. Accreditations across borders are offered by agencies such as the US 
regional accreditation commissions and by private professional organisations such as 
the US based Association to Advance Collegiate Schools of Business (AACSB) or the 
Accreditation Board for Engineering and Technology (ABET). In Europe, the European 
University Association (EUA) reviews universities, and professional organisations such 
as the European Quality Improvement System (EQUIS) review business schools, and 
the veterinary professional organisations work together to set up shared procedures and 
criteria for accreditation. 

In Portugal the EUA has been active in reviewing several universities, and among 
the professional associations, the Order of Veterinary Doctors is deeply involved in 
European cooperation on shared standards for accreditation of veterinary programmes. 

The point could therefore be made that Portuguese higher education institutions 
should be permitted to shop globally - and European - for accreditations performed 
by international accreditation operators, whether agencies of professional organisations 


or not. However, the ENQA panel considers it a very important point that the 
Portuguese government launches a new and strong quality assurance system that 
ensures that accreditation results over a broad range of fields are credible, consistent 
and comparable; thus providing essential information for young Portuguese searching 
for their preferred study place and for employers searching for graduates with the best 
educational background. 

If that is the goal, the new system should be totally informative and its results be 
comparative. Therefore, the system must firstly be based on comprehensiveness and 
the mandatory inclusion of all study programmes within the coming accreditation 
cycles. It is essential that all study programmes within a field are held to measure 
against the same minimum criteria for quality. A search for the right study place 
or the identification of the relevant graduate would be seriously hampered if three 
study programmes were accredited by the new national agency and one programme 
accredited by some international provider of accreditations with its own set of criteria. 

Secondly, the need for international transparency should be served by the anchoring 
of the new Portuguese system to the European Standards. Furthermore, the use of 
international experts, the implementation and reporting in English will ensure an 
international dimension in quality assurance processes. 

At the same time, it should be recognized that Portuguese institutions and 
programmes may perceive an added value in submitting themselves to accreditation 
by international operators. It is an evident implication of institutional autonomy that 
the institutions should be permitted to do so. But it should also be evident that in 
such a case institutions must fund the accreditations out of their own budgets and 
that the programmes in question would still have to undergo the mandatory, cyclical 
accreditations by the new Portuguese agency. Furthermore, the institutions should be 
restricted to contract with quality assurance agencies that will be recognised in the 
so-called European register. In 2005, the Bologna ministers invited ENQA and its 
partners to submit to the ministers the plan for a register of quality assurance agencies. 
The register is planned to identify and list those agencies that in terms of organisation, 
procedures and methods are fully compliant with the European Standards. 

In other words there is an argument that the future quality assurance system leaves 
room for other national and international operators, but it is even more important that 
the system ensures a high degree of consistency and comparability in processes and 
results. Therefore care must be taken to 

• restrict the use of other operators to those that are listed in the coming European 
register 

• apply the same minimum criteria for quality 

• give the board of the new national agency a mandate to validate and endorse the 
reports that are produced as a results of processes implemented by other national 
or international operators 

• give the board the mandate to refuse reports that do not comply with the 
European standards or the general quality criteria set by the national agency 

• give the board the mandate to make all accreditation decisions irrespective of 
operator responsible for the process 


The ENQA panel recommends that the new agency covers all Portuguese study programmes 
of higher education and, accordingly, that the accreditations carried out by the national 
professional associations do not need to continue. 

The ENQA panel recommends that higher education institutions and programmes have the 
formal right to let them be accredited by international accreditors, but on a self-funding 
basis and without the option of excusing themselves from the mandatory accreditations of 
the new Portuguese agency. 

The ENQA panel recommends that the board of the quality assurance agency has the 
mandate to accept or refuse reports from other national or European operators and to 
make all accreditation decisions irrespective of operator responsible for the process 

7.3 Methods and levels of a Portuguese quality assurance system 

Two issues are covered in this section, namely the number and types of methods that 
a Portuguese national quality assurance system should cover and how these are to he 
applied with regard to institutional and programme levels. 

Concerning methods, there is great variation in the scope and scale of quality 
assurance systems in Europe, as well as internationally. The Netherlands and the 
USA have ’’pure” accreditation systems, while Denmark, Austria and to some degree 
Germany employ a number of methods ranging from accreditation to academic auditing 
and evaluation. Concerning levels, the European trend is moving towards a distinction 
between accreditation at the programme level and academic audit, and evaluation at 
the institution level. 

The decision of the Portuguese government to set up a quality assurance agency 
should not preclude a combination of accreditation with, say, academic auditing at 
institution level. It is the opinion of the panel that a mature quality assurance system 
should operate at both programme and institution levels. A combination of different 
quality assurance methods will make it possible for the Portuguese government to 
include in its design of quality assurance both the element of control (accreditation) and 
the element of improvement (academic audits). 

The above argument is based on the emerging agreement among many in the field 
of quality assurance that academic standards will most likely be best assured when 
increasingly autonomous institutions have in place strong and effective mechanisms for 
improving the quality of all their academic study programmes. Therefore an important 
evaluation function of the proposed Portuguese quality assurance system would be to 
help develop the capacity of higher education institutions to assure academic standards 
and to improve teaching and student learning. 

A number of quality assurance agencies in EU countries now conduct both 
program accreditations as well as academic audits. Academic audit processes are 
particularly well developed in Denmark, Einland, Norway, Sweden, and the UK. 
Because academic audits examine an institution’s own processes for assuring the 
quality of its academic programmes, their focus can be a well functioning compliment 
to programme accreditations. The central attraction for the institutions is that they 
receive a comprehensive analysis of the level of their own internal quality assurance, 
and one on which they may base their future quality work. It follows that higher 
education institutions have, as a rule from the beginning, adopted a positive attitude 


towards academic audits, in contrast to the prevailing critical attitude towards 
evaluations in the 1990s. 

The ENQA panel recommends that accreditation at programme level be combined with 
academic audit at the level of higher education institutions. 

The ENQA panel has considered whether this recommendation should argue for two 
parallel systems: an accreditation system and an academic audit system. The panel’s 
conclusion is that one comprehensive agency is preferable, because such a quality 
assurance agency covering accreditation and academic audit would have a stronger 
financial and staff base. At the same time the agency and the institutions would all 
benefit from the synergy between accreditation and academic audit. 

The ENQA panel recommends 

• that the new quality assurance agency decides on a first cycle of accreditations at the 
programme level, beginning in the second half of 200/; 

• that this first cycle covers at least the BA level of study programmes in all public and 
private university and polytechnic study programmes; 

• that the new quality assurance agency announces a future cycle of academic audits 
and then carefully designs an academic audit process appropriate to the needs of 
Portuguese higher education to be introduced following the completion of the first 
cycle of programme accreditations; 

This schedule would allow the institutions time to adapt to the implications of the 
new system of programme accreditation as well as provide further incentive for the 
development of their own internal quality assurance processes. The academic audits 
themselves could then focus on sharing best practice, providing an external assessment 
of existing processes and recommendations for improvement as a means of advancing 
the development of institutional quality assurance processes. 

In order to regulate the number and quality of higher education institutions 
the agency should consider also introducing institutional accreditation. Institutional 
accreditation is relevant in three connections. 

Firstly, in the case where a new institution is established and applies for approval 
to offer either undergraduate or post graduate study programmes. It can be either a 
private or public institution. 

Secondly, in the case where an institution applies for an elevated institutional status. 
In Portugal as in a number of other European countries the distinction between the 
university sector and the polytechnics is being eroded, and an increasing number of 
polytechnics are introducing research activities and postgraduate study programmes. 
Therefore, it is expected that a number of polytechnic institutions will aim to elevate 
their institutional status to become universities 

Thirdly, in the case where the Ministry suspects that the programmes offered by 
a certain institution suffer from poor quality. One type of evidence could be that 
institution’s repeated failures in obtaining programme accreditation. 

In all three cases the institution should undergo an institutional accreditation that 
determines whether the institution in question has appropriate and credible structures 
and mechanisms in place to ensure the quality of its activities. The result of an 


institutional accreditation process should then be a yes/no decision. The process of 
institutional accreditation would then replace the existing ministerial approval of 
institutions. The main argument of the review panel is that institutional accreditations 
should not be organised on a systematic, cyclical basis but should take place on an 
ad hoc basis. 

In the first two cases mentioned above the initiative for accreditation should be 
that of the higher education institutions in question. Only in the third case would an 
initiative by the Ministry be the normal procedure. 

The ENQA panel recommends that the new agency consider introducing institutional 
accreditation in order to steer and regulate the number and quality of the higher education 
institutions. 

7 . 4 . One strong national agency 

A widely acknowledged weakness with the existing quality assurance system is the 
complexity and inconsistency in the system. 

The panel’s recommendation for a future quality assurance system is a single one-tier 
system. As argued in section 7.3, this agency should be responsible for accreditation 
of study programmes and institutional audit of all higher education institutions. The 
agency should have the following characteristics: 

The authority to make accreditation decisions 

The accreditation decision should be entirely based upon an assessment of the academic 
quality of the study programmes resulting from the accreditation process. The decision 
to launch new study programmes according to criteria for national relevance should 
be vested in the Ministry. The dual decision procedure will be further elaborated in 
the next section. 

Vesting the authority to decide on accreditation in the quality assurance agency 
contributes to creating a high level of transparency as far as criteria and the procedure 
are concerned. The drawback is that the same body is responsible for both the process 
and the decision that follows from the process. To make an allowance for this, it is 
necessary to establish a well-functioning appeals system for the benefit of the accredited 
study programmes. The issue of an appeal system is dealt with in 7.22. 

In order to reduce the risk that a decision to launch an accreditation or set up an 
accreditation cycle is mingled with political interests or motives, however, it maybe an 
obvious solution to vest the right of the initiative in the board of the quality assurance 
agency. 

The board 

The agency should be truly independent of the government, higher education 
institutions and other parties, as stressed in the European standards. An independent 
board with members appointed in their personal capacity should guarantee the 
independence of the agency. 

The board should be small and comprise approximately ten members. The board 
should meet frequently (once a month) and should have the authority to initiate 
programme accreditations and institutional audits on a cyclical basis, approve the 


external experts and the composition of the panels, and, in accreditation cases, decide 
whether a programme meets the specific quality criteria or not. 

The membership of the board should reflect established professionalism in 
overseeing quality assurance processes and understanding the process of accreditation 
decisions. However, the panel wishes to point to the possibility that the board 
establishes itself with subcommittees representing sector specific knowledge. Such a 
construction could contribute towards enhancing credibility. 

The integrity and independence of board members must rest on the fact that they 
are all appointed in their personal capacity. This prerequisite should prevent a situation 
where board members could bring their institutional interests into board decisions. 

There must be continuity in the work of the board, but membership of the board 
attaining semi-permanent status should be avoided. Members should therefore be 
appointed for a three-year period, with the possibility of this being prolonged once. The 
members should be appointed on staggered terms in order to avoid the whole board 
changing over the same year. This leads to the key issue of the procedure for appointing 
board members. 

It is a central lesson from CNAVES that the composition and affiliation of an 
agency’s governing body is essential for the way in which institutions, stakeholders 
and government perceive the agency’s degree of independence. The ENQA panel has 
considered various models based on their international experiences. The conclusion is 
that the government should appoint as chairperson a respected member of Portuguese 
society, but without any present employment, either in a higher education institution 
or in government. 

The government further should appoint ordinary board members based on 
the nominations from the constituent member groups in the advisory council 
recommended below. 

The government may at its discretion also appoint two or three non-Portuguese 
members with international experience in quality assurance of higher education. 
Nominations for these members could naturally be invited from the board of ENQA. 

An advisory council 

To ensure a wider involvement of stakeholders in, the board should be supplemented 
by an advisory council with representatives from relevant stakeholders, including 
higher education institutions, employer organisations, professional associations, student 
unions and perhaps international experts. The advisory council should not have any 
formal decision making authority but should provide advice to the board on issues 
related to the quality assurance of higher education. 

The ENQA panel recommends establishing an agency with clear responsibilities for quality 
assurance and accreditation. An independent professional board appointed by government 
with decision-making authority should guarantee the independence of the organization. 

In addition, an advisory council should be established with a broader representation of 
relevant stakeholders. 

7.5 Independence of the agency 

The independence of quality assurance is a key element of the European standards. The 
independence of quality assurance agencies concerns independence from ministries 


as well as from higher education institutions. There is no doubt that the most often 
mentioned critique of the existing quality assurance system is the lack of independence 
in relation to higher education institutions. 

The European standards highlight the issue of operational independence as the 
most important aspect of independence. This implies that definition and operation 
of procedures and methods, nomination and appointment of external experts, and 
formulation of conclusions and recommendations in the reports are undertaken 
autonomously and independently from ministries, education institutions and other 
stakeholders. 

The operational independence should be ensured through official documents, e.g. 
a legislative act, which is the case in most European countries. The legal bases of the 
quality agencies in Europe guarantee to different extents the operational independence 
of the agencies. 

It is important that the mandate of the new quality assurance agency is clearly 
described in the legal framework. An unclear mandate can open a window for 
ministries, institutions, and other kinds of organs with political interests to influence 
operations. On the other hand, the legislation must not be too detailed, for instance 
in its specification of the criteria that form the basis for the accreditations. The 
formulation of the assessment criteria is a process that should be vested in the quality 
assurance agency in order to ensure that the criteria are continuously adjusted and 
revised in accordance with actual quality developments that take place at the higher 
education institutions. 

The point here is that, on the one hand, the legal framework should ensure that 
the mandate of a new quality assurance agency is clearly defined to protect the 
agency from political pressure and opportunistic behaviour. At the same time, the legal 
framework should be formulated in a way that allows a certain degree of flexibility 
in the management and operation of the accreditation system, e.g. determination of 
criteria, adjustment in procedures, etc. The flexibility is needed to ensure that the 
accreditations do not promote conservatism and rigidity in the quality assurance of 
higher education. 

The ENQA panel recommends that the operational independence of a new quality 
assurance organisation in Portugal be ensured through a legal framework that clearly 
describes the mandate and tasks of the quality assurance agency, while providing the 
necessary flexibility for the operational level to make necessary adjustments to operational 
procedures. 

In most European agencies an independent board or council is perceived as 
the major guarantee of independence. Section 7.4 above contains reflections and 
recommendations on the role of an independent board for the new agency. 

7.6 Data and information system 

A common concern is the inadequacy of information relevant to the improvement of 
academic programmes, particularly information relating to programme graduates, the 
labour market and employees. In a mixed and competitive system of higher education, 
such as in Portugal, the provision of valid information on academic programmes can be 
a critical component of an effective quality assurance framework. 


The European standards have as a key element the need for higher education 
institutions to posses elaborate and operational information systems. A number of 
countries are also attempting to define the essential academic programme information 
that should be publicly available to ensure the effective functioning of their quality 
assurance systems. This issue is currently being debated in Australia, the UK, and the 
US. While a number of these initiatives are motivated by a desire to better inform 
students on choice of institution or study programme, the more significant contribution 
of these data may be to strengthen the capacity of institutions to improve academic 
programmes. The collection and publication of these data reveal quality variations 
among academic programmes, thereby helping institutions discover best educational 
practices, as well as identifying programmes in need of improvement. Encouraging such 
a “culture of evidence” is often an essential first step to motivating effective collective 
actions for self-improvement within academic communities. 

The ENQA panel recommends that an early project of the new agency be to ensure 
that information for each study programme needs to be publicly available and to make 
recommendations to the Ministry regarding the best means of providing these data. 

There are various approaches to providing the necessary data. Academic institutions 
may be required to produce and publish the data themselves, and accreditation reviews 
may be required to verify the accuracy of the relevant data. A government agency 
may collect the necessary data and publish the reports. Alternatively, third parties 
may provide the necessary data under contract with individual institutions or with the 
government. 

7.7 Cross-sector reviews 

The European standards emphasise that quality assurance agencies should produce 
summary reports describing and analysing the general findings of their reviews, 
assessments and evaluations. The rationale for this recommendation is that quality 
assurance agencies collect a wealth of information about individual programmes and 
institutions that provides material for structured analyses of the entire higher education 
system. Such analyses about developments, trends, emerging good practice and areas 
of persistent difficulties can become useful tools for policy development and quality 
enhancement. 

The ENQA panel recommends that the new agency establish a procedure for producing 
summary reports every second or third year. 

7.8 Professionalism and consistency in the work of the quality assurance 
agency 

The credibility of external quality assurance processes depends on a high professional 
level in the agency responsible. The level of credibility is also closely linked to 
the consistency between processes. Therefore, the European standards include a 
recommendation that quality assurance agencies should have adequate human and 
financial resources to match the tasks they are required to perform. This is not the 
case in the existing quality assurance system where the decentralized organization 


made it very difficult for CNAVES to develop and utilize a professional staff in quality 
assurance, as is the common characteristic of effective agencies in other EU countries. 

In other words, there must be in the new agency a staff, which both in terms of 
number, qualifications and resources is capable of ensuring methodical professionalism 
and consistency of the individual accreditation processes and from one accreditation to 
the next. This concern is especially important in relation to programme accreditations, 
which may otherwise suffer from the fact that different expert panels are going to 
view various elements differently and will make their own independent interpretation 
of the criteria. 

The professional staff of the agency should ensure therefore the consistency in 
the operational processes and reporting in relation to accreditations and audits. 
However, considering that in the current model of CNAVES and the four evaluation 
councils the processes and reporting are the responsibility of the external experts this 
recommendation absolutely implies a role reversal in the new system for the external 
experts in relation to the staff of the agency. 

The external experts should have the responsibility for the academic assessments of 
the accreditations and audits and the resulting conclusions and recommendations. But 
the responsibility for the applied methodology, the quality assurance process and the 
drafting of the report should be vested with the professional staff of the agency. This 
division of labour would reduce the risk of inconsistency in the interpretation of the 
criteria and in the reporting from one accreditation or audit to the next. 

A similar division of responsibilities between review panels and professional staff is 
in place in a large number of established national systems. This has not been the source 
of conflicts and on the contrary it is the Danish experience that recruitment of panel 
members is substantially eased by the argument that experts are not expected to spend 
weekends or holidays drafting reports. 

In Denmark the division of responsibilities is included in the formal contract signed 
by the individual panel members and EVA’s executive director. 

It should be added that a strong professional staff will be able to secure the necessary 
methodological innovation of the agency; it will be able to organise proper processes; 
and it will be in a position to form the necessary links and networks with other 
European or international quality assurance agencies. 

One of the first tasks of the board of the new agency must be to consider and 
define the relevant profiles of staff members. In this task it is important that focus 
is not exclusively on recruiting staff members with high academic credentials and 
seniority. Being a good professional in the management of quality assurance depends 
on additional qualifications than that of a PhD in an academic held. Professional staff 
members must have strong analytical resources and the capacity to work in teams. 
Strong qualifications in English will be necessary, partly because review processes will 
involve English speaking experts, and reports must be drafted in English, and partly 
because skill in written and spoken English will be the entry ticket to participation in 
European and international cooperation. 

Additionally, it is important to secure staff members with high qualifications in 
applied methodology, qualitative as well as quantitative. Special attention should also 
be given to establishing the necessary expertise in communication and information 


services, so that the new agency experiences greater success than CNAVES in making 
processes and results transparent and publicly available to Portuguese society. 

The panel does not wish to be unduly explicit on the qualifications of the executive 
head of the new agency. The priority must be to recruit an executive whose profile is 
one of the highest personal integrity and who will be respected in the higher education 
system. Political acumen and sensivity is also necessary, but must be combined with the 
knowledge and flair with which to build up a strong organisation with committed staff 
and enjoying the trust of the higher education community. 

In the establishment phase the new quality assurance agency would benefit from 
the use of external consultants with relevant international experience to advice on 
organisational and methodological matters. 

The ENQA panel recommends establishing an agency with a strong and highly qualified 
staff and with a clear division of responsibilities between external expert panels and the 
staff 

The ENQA panel recommends further that an executive director be appointed with a strong 
personal profile and relevant experiences in organisation building and quality assurance. 

The ENQA panel recommends further that the quality assurance agency be established on 
the basis of a budget, which provides for a professional staff consisting of not less than 
twenty-five qualified members. 

The ENQA panel recommends that in the establishment phase the quality assurance 
agency cooperates with external consultants with international experience to advice on 
organisational and methodological matters. 

7.9 Use and role of experts 

Considering the prevailing criticism of the external experts appointed in the existing 
quality assurance system, it is of course important that the new quality assurance 
agency sets up procedures for the recruitment of experts that secure academic expertise 
at a high level, coupled with personal integrity and independence of institutional or 
governmental interests. Integrity and independence are the keys to the credibility for 
which the new agency must be recognised by the higher education institutions and 
other stakeholders. This point is also reflected in the European standards and leads to 
a recommendation, which should of course be seen in the context of the arguments and 
recommendation in section 7.10 on international experts. 

The ENQA panel recommends that the quality assurance agency establishes and enforces 
a no-conflict-of-interest mechanism in the appointment and work of its external experts. 

The panel observes that in the existing quality assurance system there is a marked 
tendency to equate expertise and integrity with the appointment of elderly, mostly 
male, full professors from the public universities. The panel does on its part consider 
it very important that the national Portuguese experts appointed represent a fair 
proportion of younger and/or female qualified academics in order that experiences of 
quality assurance gained by experts create a positive spread effect in higher education 


institutions. Neither does the panel support the argument it met during several 
interviews that integrity and independence of interests necessarily increase with age. 

The ENQA panel recommends that experts are appointed through a process that ensures a 
balanced age and gender profile and an appropriate insight into quality assurance. 

In the existing quality assurance system inconsistency in the focus and conduct of 
the external expert teams is perceived to be a common problem. CNAVES points in its 
self-evaluation to the fact there was never an adequate level of training of experts. The 
reasons given for this are lack of funding and organisation, combined with a low level of 
motivation among the experts to attend the training that was in fact offered. 

The ENQA panel recommends that the new quality assurance agency receives the necessary 
funding to organise systematic training of experts for the review panels. 

At the same time, the panel recognises that if the agency staff is sufficiently strong in 
terms of process-administration and methodology, the training of experts could focus 
primarily on preparing them to conduct themselves professionally and efficiently during 
site-visits. 

7.10 Use of international experts 

It is an element of the European standards that European external quality assurance 
systems should be encouraged to open their processes as much as possible to 
international expertise. In this perspective a definite weakness of the external reviews 
in the existing quality assurance system is the heavy reliance on reviewers from 
Portugal, even though CNAVES asserts in its self-evaluation report that a measurable 
percentage of international evaluators are utilized in the reviews, and that this 
increased over time. CNAVES also notes that it did not have the authority to require of 
the institutions that reviews were conducted in English, which would be necessary in 
the case of international teams. 

Private institutions were particularly outspoken on the need for fair balance in 
panels and for international reviewers. They noted that almost all of their external 
reviewers came from the Portuguese public higher education institutions (i.e. their 
competition) and often appeared biased against private study programmes. 

Finally, while there was strong support for international reviewers, there was also 
support for CNAVES’ viewpoint, namely that members of external review teams be 
knowledgeable about Portuguese culture and the national system of higher education. 
The panel notes that the international teams successfully used in the research 
evaluations were also sensitive to this issue. That is, these teams were chaired by 
Portuguese researchers, although the latter were not permitted to vote on the team’s 
evaluations. 

The panel recommends that members of external review panels typically number four-five 
members. Of these, at least two members must come from countries and or higher 
education institutions outside Portugal. At least one member must be from Portugal, but 
not represent a Portuguese higher education institution. 


In many European quality assurance systems the inclusion in review panels of 
international experts is now being introduced. The major obstacle is in most cases 
the added resources involved in basing review processes and their documentation on a 
non-national language, because international experts do not speak the local language. 
One solution to this problem could be to appoint, if possible, Portuguese academic 
expatriates as experts, but in that case the recommended procedure should be that 
such experts have not been working in Portuguese higher education institutions for at 
least five years. 

In any case, the use of international experts will add to the costs and burden of the 
quality assurance system, but it was evident from the interviews with representatives 
from higher education institutions on the site visit that the burden, if not necessarily 
the cost of this change, would be accepted as quid pro quo for increased credibility 
and better experts. 

The ENQA panel recommends that the funding of the new quality assurance agency is 
at a level which makes it feasible to conduct reviews in English or another international 
language. 

7.11 Inclusion of students in the quality assurance process 

The extent of student participation in the quality assurance processes in higher 
education institutions presents a differentiated picture across Europe. Participation 
may take place in three contexts: Firstly, as part of the process at the institution or 
programme under review; secondly as members of review panels; and thirdly, through 
representation on boards/councils. 

Students will in most cases be represented in the self-evaluation process. 
Nevertheless, they do not seem to influence these processes in any significant manner. 
The impact of the students will therefore in most cases be linked to interviews with 
student groups during site visits. This state of affairs has also been characteristic in the 
existing quality assurance system where the role of students in the quality assurance 
processes has been unduly reduced. As is the case in most European countries at 
present, students have not been invited to join review panels. 

The European students’ organisation ESIB had an important role in the work leading 
to the ENQA report for the European ministers in Bergen. All involved parties accepted 
the position of ESIB and that the European standards should stress the importance of 
student participation as well as possible panel members in quality assurance processes. 

The ENQA panel recommends, therefore, that the new quality assurance agency firstly 
ensures that it is an unconditional element in the self evaluation processes and the 
site visits that student representatives have a roll that can lead to actual contribution 
to the process. Secondly, the new agency should set up procedures that ensure that 
appropriateness of including students is considered when external review panels are 
composed. 

It must be recognised that the process of identifying student members of review panels 
is not necessarily an easy one. Care must be exercised that student members introduce 
the specific interests and concerns of students placed within the study programme into 
the work of the panels. They should be accepted therefore as panel members in their 


personal capacity rather than as spokesmen for the general political platforms of the 
national student organisations. 

Student participation should also be considered in relation to the advisory council 
of the new agency. In case of participation, student members should, in contrast to 
the argument above concerning panels, be able to articulate in the advisory council 
more general student related quality assurance policies. There is a significant number of 
student organisations, but these could be invited to nominate members to the advisory 
council to the board. 

The ENQA panel recommends that the Portuguese government appoint representatives of 
student organisations to the advisory council. 

7.12 Stakeholder Involvement 

A deficiency in CNAVES’s evaluation process is the lack of active participation of 
relevant stakeholders in quality assurance processes. In other European countries, 
employer representatives in particular are invited to join expert panels to bring 
in an external perspective that supplements the perspectives of experts from the 
higher education system. Another applied approach used especially in the Danish 
quality assurance system is the inclusion of user surveys as an important source of 
documentation to complement and verify the self-evaluation documentation of the 
institutions. The user surveys will typically cover employers or graduates. 

The ENQA panel recommends that the view of relevant stakeholders, e.g. employers and 
graduates, be taken into account in external quality assurance processes. 

The following approaches could support this recommendation: 

• Accreditation review panels could include at least one member with a qualification 
as employer of graduates from the programme 

• Higher education institutions could be encouraged to establish within the major 
programme fields permanent panels of employers and/or recent graduates. Partly 
these panels could serve as running feedback to institutions on the quality of the 
programmes in question. Partly the panels could contribute directly with their 
observations to external quality assurance processes. 

• In connection with accreditation processes higher education institutions could 
be encouraged to initiate quantitative surveys of employers’ or recent graduates’ 
impressions of quality of the programmes under accreditation. This approach 
would be especially relevant in the cases where programmes face substantial 
employment problems for their graduates. 

7.13 International cooperation and networking 

A priority must be to ensure the international significance and relevance of the 
Portuguese quality assurance system. Here, active participation in European and 
international cooperation is a very important tool. The highest priority should go 
to achieving membership of the ENQA where there is a continuous sharing of 
knowledge of best practices. In several European countries, a process is now 
well under way towards translating the European standards into the respective 
national quality assurance systems and processes. A shared methodological platform 


and shared standards are very important elements towards assuring international 
transparency, comparability and compatibility between individual national quality 
assurance processes. 

Another potentially relevant framework for cooperation could be the somewhat 
more exclusive European Consortium for Accreditation (ECA) where seven European 
national agencies work together towards harmonisation of their accreditation processes 
with the aim of establishing mutual recognition of accreditation results. 

At the global level, the International Network of Quality Assurance Agencies 
in Higher Education (INQAAHE) is the obvious platform for introducing the new 
Portuguese system internationally. 

However, active international and European networking does put a strain on a 
quality assurance agency’s resources, human as well as financial. It is, at the same 
time, important that international activities are not exclusively the domain of the top 
management of the agency, but also include ordinary staff members, so that these may 
have the opportunity to improve their own quality assurance professionalism through 
meetings and workshops with international colleagues. Another relevant instrument is 
staff exchange or trainee periods at other agencies. The Danish agency has had staff 
exchanges twice with the Swedish agency and has over the years received trainees from 
several European agencies. 

The ENQA panel recommends that the new quality assurance agency be given the directive 
and the means to engage actively at all levels of the agency in international and European 
cooperation and networking. 

7.14 Internal accountability procedures and compliance with the European 
standards 

A core element in the European standards is the requirement that external quality 
assurance agencies must have procedures in place for their own accountability. The 
background for this standard was an ENQA process from several years ago that was 
focused on the idea that agencies must be prepared to ’’take their own medicine” 
to quote the title of an ENQA workshop on this subject in 2004 in Barcelona. The 
ENQA membership standards specify accordingly what elements must be included in 
the internal quality assurance of agencies. 

According to the European standard on this issue, agencies are expected to have a 
published policy for the quality assurance of the agency itself, and made available on its 
website. This documentation must, along with other documentation, demonstrate that 
the agency’s processes and results reflect its mission and goals for quality assurance; 
that the agency has internal quality assurance procedures in place which include an 
internal feedback mechanism (i.e. means to collect feedback from its own staff and 
council/board); an internal reflection mechanism (i.e. means to react to internal and 
external recommendations for improvement); and an external feedback mechanism 
(i.e. means to collect feedback from experts and reviewed institutions for future 
development) in order to inform and underpin its own development and improvement. 

Einally and most importantly, the agency is required to undergo a mandatory cyclical 
external review of its activities at least once every five years. The results of the cyclical 
external reviews lead for one thing to the decision of ENQA as to whether the agency is 
- or continues to be - fit for membership. Just as important, however, the results should 


form the basis for the decision as to whether the agency can be listed in the coming 
European Register as an agency complying with the European standards. 

The European Register was introduced in the ENQA report to Bergen. The task of the 
register is to invite external quality assurance agencies operating in Europe to be listed 
according to their compliance with the European standards. In the communique from 
that meeting Ministers say that the register must be formalised and established through 
the work of ENQA in cooperation with the European organisations of universities 
(EUA), polytechnics (EURASHE) and students (ESIB). The discussions among these 
four organisations are ongoing and will end with a recommendation to the Bologna 
ministerial meeting in May 2007. The exact format and rules of the European register 
are therefore not yet determined. 

The standard on internal accountability is perhaps in the full scope of its 
requirements that standard which will demand the greatest efforts for many established 
European agencies to comply with. The panel realises this fully and understands that 
the new quality assurance agency needs proper time to develop its own accountability. 

If the new agency is established by the beginning of 2007, the first mandatory external 
review must take place by 2012 at the latest. 

It is the view of the panel that a full five year period is a rather long period to 
wait for a new agency before it achieves learning feedback and corrections from an 
external review. 

The ENQA panel recommends that a mid-term review of the new agency takes place by 
the end ofioog. 

This review can identify the extent to which the agency at that time complies with the 
European standards and provide advice and feedback on how to achieve full compliance 
prior to the end of 2012. 

The ENQA panel recommends that the new agency gives a high priority to a process leading 
towards full compliance with all the European standards and thus secures its placement 
in the coming European register of external quality assurance agencies at the very latest 
by the end of 2012. 

7.15 Pre-accreditation, accreditation and relevance 

Recent years’ explosive growth in the number of new study programmes in Portugal, 
caused particularly by the extended autonomy of the public universities and their ability 
to propose and subsequently launch new study programmes, calls for a more effective 
national steering and the coordination of the provision of new study programmes in 
Portugal. Pre-accreditation could be an effective instrument to ensure that the approval 
of new study programmes is based on a proper quality assessment. The process of pre- 
accreditation would in the case of private institutions replace the current Portuguese 
system of ministerial approval of new study programmes. 

The process of pre-accreditation should be placed in the same agency that is 
responsible for the accreditation process of study programmes (ex-post) to create a high 
degree of transparency and consistency in the preliminary screening of the quality and 
in the subsequent assessment of the quality of existing study programmes. 


Accreditation (first cycle) is carried out three to four years after the initial approval 
to ensure that existing study programmes continue to live up to declared requirements 
and criteria. 

However, the issue of national relevance, e.g. employment possibilities, student 
recruitment patterns, duplication of study programmes among institutions, is a process 
that should be vested in the Ministry and separated from the academic assessment. 
Providing a positive pre-accreditation (the board of) the quality assurance agency can 
recommend the approval of a new programme to the Ministry which can subsequently 
conduct a screening of the programme according to a set of pre-defined criteria for 
relevance. 

The ENQA panel recommends: 

• introducing a two-step accreditation process of pre-accreditation and ex-post 
accreditation in order to create a high degree of transparency and consistency in the 
two processes; 

• that the final decision for provision of new study programmes be vested within the 
Ministry based on criteria for relevance and national concern. 

7.16 Financing of the new quality assurance agency 

It follows from several of the recommendations for the new agency that this must 
be financed at a level that, among other issues, ensure adequate professional staff, 
appointment and training of (international) experts and active participation in global 
and European cooperation. Appendix VII includes a budget for the estimated running 
costs of the agency, and the establishing costs. 

There are various approaches for the financing of the recommended quality 
assurance agency. Financing could in principle be a government responsibility. A 
hundred percent government-funded model is applied in the Nordic Countries and in a 
number of European countries that have a primary focus on programme accreditation. 

Another approach is a co-financing model where part of the costs of running 
operations should be split between government and higher education institutions. 

The consideration of co-financing could be based on two principles. One principle 
could be a fixed ratio between the contribution of the quality assurance agency and 
the higher education institutions. If this principle is applied the higher education 
institutions could be asked to contribute a maximum of fifty percent of the full costs 
of the process. This contribution from the higher education institutions should cover 
the direct costs for running an accreditation process including salary and other costs 
that are directly linked to the operation processes. The contribution from the higher 
education institutions should thus not cover the general costs of the agency. If this 
approach to co-financing is followed the fifty percent contribution could be gradually 
introduced with a lower level in the beginning, for instance 30 percent, rising to 50 
percent in the next cycle. 

Another principle for co-financing could be a variable ratio between the contribution 
of the quality assurance agency and the higher education institutions. The primary 
parameter could then be size of the institution in terms of the total budget and 
student enrolments of the programmes offered. This principle could be realised in two 
ways: One model is an overall fee that the institutions pay in order to be part of the 


accreditation system. Another model is a fee payment that is linked to every single 
accreditation. 

The ENQA panel recommends this second principle for co-financing is followed and that the 
first model is applied in order to minimise the administration load for the quality assurance 
agency. The degree of variation according to the size-parameter between the highest and 
lowest possible contribution from the higher education institutions could then be settled 
between a 25 and 50 percent contribution. 

Accreditation of new programmes could be based on a full fee from the institutions. It is 
the assumption of the panel that a model of institutional self-financing will regulate the 
number of new programmes submitted for approval via a process of pre-accreditation 
and thus avoid a continuation of the recent years’ explosive growth in the number of 
new study programmes 

The ENQA panel recommends that accreditation of new programmes is based on a full fee 
from the institutions. 

7.17 The focus of the accreditation criteria 

The criteria for programme accreditation should be formulated at two levels. One level 
concerning generic criteria and one level that focuses on area specific criteria. The area 
specific standards should, where relevant, take into account criteria that are applied 
internationally, e.g. those applied by the Order of Veterinary Doctors. 

The generic criteria should function as minimum requirements that apply to all 
types of study programmes and relate to the two cycles of bachelor and master. 

Thus the generic criteria should be formulated within the framework of national legal 
requirements and the Bologna process that specifies the outcome requirements for 
bachelor and master levels respectively. 

The reference framework for the generic criteria should be the European framework 
for qualifications. The framework has been developed by a Bologna working group 
on a qualifications framework and comprises an overarching framework with a high 
level of generality. The framework includes cycle descriptions in the form of generic 
qualification descriptions that should be used as a common framework across Europe. 
The so-called Dublin-descriptors developed by the Joint Quality Initiative have formed 
the background for the cycle descriptors for the framework. They offer generic 
statements of typical expectations of achievements and abilities associated with awards 
that represent the end of each Bologna cycle. 

The area specific criteria on the other hand should focus on the characteristics of 
specific study programmes within a certain study field, e.g. engineering, economy, etc. 
The criteria should be comprehensive, covering input, process and learning outcomes, 
and comply with the European standards. When relevant, the criteria should take into 
account criteria applied internationally by professional associations in the respective 
fields, e.g. engineering, economy, veterinary doctors, etc. 

Due to the different requirements for universities and polytechnics, separate criteria 
for each type of institutional programmes should be formulated. Appendix VIII includes 
a proposal for such differentiated criteria. 


The ENQA panel recommends: 

• that programme accreditations he based on generic as well as area specific criteria in 
order to provide a comprehensive picture of the quality of the study programmes; 

• that differentiated criteria are formulated to reflect the characteristics of study 
programmes offered at universities and polytechnics, respectively; 

• the application of the European Qualification Eramework in the definition of generic 
cycle-based criteria; 

• that the area specific criteria cover input, processes and learning outcome that comply 
with the European standards, and when relevant take into account criteria applied 
internationally by professional associations in the respective fields. 

7.18 The process for defining the accreditation criteria 

Another key question is who should he responsible for defining the accreditation 
criteria? Should it he the Ministry, the quality assurance agency, the professional 
associations or the education institutions - or should they he formulated through 
interaction or actual cooperation between the parties? 

A crucial element in this connection may be whether the final decision on the 
content and delimitation of the criteria is made sufficiently independently of the 
education institutions. 

Another important consideration, as stressed earlier in the chapter, is to ensure that 
there is sufficient flexibility and broadness in the legislative framework to allow further 
development and adjustment of the criteria over time, in turn to ensure that the criteria 
reflect both the continuous quality development at the institutions and the academic 
development of the study programmes. 

The ENQA panel recommends: 

• that the criteria be formulated through appropriate interaction between the 
legislative/framework-setting and the operational agency level, and that this process 
guarantees sufficient independence in relation to the education institutions involved; 

• that the criteria for professional programmes be developed in close cooperation with 
the professional associations, taking into account the criteria that already exist. 

In addition, thought should be given to the element of innovation of criteria. One of the 
dangers of any accreditation system is that of ossification of criteria. The unfortunate 
result may be conservatism in accreditation decisions leading, for instance, to non- 
accreditation of new untraditional, innovative or cross disciplinary study programmes. 
An import priority is, therefore, flexible and broad legislation that opens up for further 
development and adjustment of the criteria. 

7.19 Categories of accreditation decisions and follow-up 

The number and types of categories of decisions is yet another important issue where 
different practice exists. Is it to be exclusively a yes/no option, as in the USA and the 
Netherlands? Or should it be possible to award conditional accreditation allowing the 
programme the possibility of meeting unfulfilled criteria within a specific deadline and 
subsequently being awarded positive accreditation? 

International experience to date makes the answer to this question reasonably 
obvious. The consequence of a sharp yes/no option is that over time the system will 


lose its credibility. The reason is that accreditation agencies - expert panels - shy away 
from giving a “no”, because the implications and consequences would be so grave for 
the programme or institution concerned. So far, the American and Dutch experience of 
accreditation clearly demonstrates this tendency. 

Another point is that an accreditation process should result in a positive added value, 
such that study programmes or institutions are motivated to transform the process 
into a tool for continuous improvement, and as a management tool. This motivation 
would be strengthened when there was a case for a conditional accreditation instead 
of a final “no”. 

The ENQA panel recommends that an accreditation system includes all three categories 
of decision, and that the credibility of the agency is consciously supported by the use of 
all three categories. 

In the case of conditional accreditation, the agency will define a time frame 
and deadline within which the programme concerned must provide the further 
documentation to establish that it meets the criteria. This procedure secures the basis 
for renewed assessment and accreditation by the agency. 

It must be a key goal that there is the necessary consistency and uniformity 
between the first accreditation and the ensuing renewed accreditation. Therefore, the 
operational, methodological and professional responsibility for the first and for the 
renewed accreditation process should be vested in the same agency. 

The ENQA panel recommends that the quality assurance agency carries the responsibility 
for follow-up on conditional accreditation. 

7.20 Consequences of accreditation 

The defining characteristic of accreditation processes is that they are concluded by 
the making of an authoritative decision on compliance, partial compliance or non- 
compliance with criteria. From this follows that a key issue is how far-reaching the 
consequences of the individual accreditation decisions should be. 

The consequences of awarding a programme accreditation should ensure public 
financing and grants to the students. Accreditation should also give the programme 
provider an added position of strength in the global market and thus attract researchers 
and students from abroad. 

If an institution over a period of time achieves a generally satisfactory record in 
programme accreditation, that institution should be entrusted by government with a 
higher degree of freedom in a number of ways, e.g. be allowed to provide education 
programmes within certain areas or at some specific levels. 

Refusal of accreditation will involve a number of consequences - typically loss 
of financial grants or loss of the right to provide specific study programmes. The 
consequence of a conditional accreditation is dealt with in 7.19. 

The ENQA panel recommends that the consequences of accreditation are given a clear legal 
basis and that practical implications for higher education institutions are clearly spelled 
out by the new agency. 


Irrespective of the combination of rewards and sanctions in any accreditation 
system, it is furthermore very important to be aware of the risk that a culture 
of compliance towards accreditation processes may emerge in the higher education 
system. The panel considers here the possibility that some higher education institutions 
will concentrate on complying with the criteria of accreditation only at the very 
minimum level required. Such a strategy should not be rewarding for institutions, 
either in the short or long term. 

The issue here is a basic implication of accreditation processes. If these are to be 
systematic and cost efficient, the focus on a minimum criteria approach is most often a 
consequence. However, accreditation reports could be tailored to make evident quality 
results well above the minimum thresholds and to highlight which programmes have 
contributed with significant examples of best practice for quality. 

The ENQA panel recommends that the new accreditation processes are planned in such a 
way that higher education institutions are encouraged to respond to the criteria applied at 
the highest level appropriate. 

One strategy to be considered in this respect could be that the accreditation reports 
include a section that recognise in a qualitative manner to what extent individual 
programmes have excelled at a higher level than the minimum thresholds. 

Under all circumstances, the panel trusts the higher education institutions to 
consider the desired quality level of their programmes within the overall design of the 
respective institution’s strategy and development. 

7.21 Length of the accreditation cycle 

The question of the duration of a time interval or cycle between accreditations should 
be viewed within the context of two overriding concerns and their mutual relationship: 
First of all, the burden placed by the accreditation process on the institutions and 
their study programmes should be taken into consideration. This concern favours a 
longer cycle. Secondly, it is important to keep in mind the value of accreditations as a 
mechanism whose credibility, in terms of continuing institutional quality development 
and the academic development of study programmes, must be apparent. This concern 
favours a shorter cycle. 

The ambition for a Portuguese accreditation process should therefore be that the 
proper balance is achieved between concern for the burden and concern for the 
credibility of accreditations. Other European agencies have over the years been faced 
with the same challenge, and the resulting best practice is reflected in the following 
recommendation. 

The ENQA panel recommends that the length of the cycle be no less than five years, and 
no greater than six years. 

In order to cover all study programmes within a cycle of five to six years, clustering of 
individual study programmes into larger discipline areas or study fields is one of the 
first necessary tasks for the new agency. For instance, physics study programmes at 
university level must all be clustered together for accreditation. The panel realises that 
grouping may be difficult because physics and other study programmes appear in many 


different variations, but this problem has been solved in other European countries. 

For example, the Danish evaluation cycle of the 1990s was based on approximately 60 
discipline areas covering almost all existing study programmes. 

Another more serious concern that should be taken into account is the preserving 
effect on study programme “thinking” that can be caused by a fixed model of clustering. 
Ultimately, the risk is that new cross-disciplinary study programmes are prevented from 
being developed. The panel is, however, convinced that this risk can be reduced by 
paying continuous attention to the issue. 

7.22 The need for an appeals forum for accreditation decisions 

When ENQA, in cooperation with its European partners from higher education 
institutions and student organisations, drafted the proposal for the European standards 
it was considered a very high priority that any quality assurance system provided 
an appeals forum allowing the higher education institutions the formal possibility 
of appealing against decisions. It is particularly crucial that a system involving 
accreditation is characterised by a due process and appeals approach, as a negative 
accreditation decision will have profound consequences for the institutions involved. 

The ENQA panel recommends that the formal framework and jurisdiction of an 
independent appeals forum be an element in a Portuguese accreditation system. 

In the work process of preparing the legislation for the quality assurance agency, 
attention must be given to the following issues: Firstly to secure the necessary 
high degree of independence; secondly to define criteria for the nomination and 
appointment of members of the appeals forum; and thirdly to define the specific 
procedures for appeals. 


Appendix I 

ENQA review of the accreditation and quality assurance practices in the 
Portuguese higher education 

TERMS OF REFERENCE 

INTRODUCTION 

In the Bologna Declaration, the European Ministers emphasised the need to 
improve co-operation in higher education quality assurance. The Declaration mentions 
co-operation in the field of quality assessment within Europe. More and more countries 
are adopting accreditation as a tool to enhance transparency. Eollowing the Bergen 
Conference, in May 2005, the Bologna Process considers a convergence of quality 
assurance practices in order to achieve the goals determined by the signatory States. 

An emerging question with the Bologna process is that it steers the Portuguese and 
other European governments towards a common solution in formal terms that may find 
little support in the realities of national strategies towards quality assurance. One of 
these realities may be linked to the remarkable growth in recent years in the fields 
of transnational education and of what is termed new means of delivery: distance 
education programmes, branch campuses, franchises and more. The identification of 
relevant strategies is going to be a challenge in the near future. A list of possible 
scenarios could include: 

• National strategies with an emphasis on regulation of importers or exporters of 
education. 

• International or regional strategies based either on supra national quality 
assurance or on meta-recognition of established national agencies. 

• Multi-accreditation implying either international recognition of national 
evaluation organisations and education structures or national recognition of a 
foreign accreditation organisation. 

The Portuguese Ministry of Science, Technology and Higher Education (MCTES) 
commissions this review exercise, which does not conflict with the responsibilities 
awarded to the higher education institutions and to the quality assurance agencies 
under the national evaluation law (Law 38/94, 21 November; Article 9, n. 3). 

This exercise was designed to ensure that the higher education system and its 
stakeholders gain maximum benefit from comprehensive evaluations by teams of 
experienced international experts and that the procedures and processes in place in 
the Portuguese higher education system can be benchmarked against best practice 
internationally. 

The review is taking place in the wider context of: 

• the Government’s strategic objective to guarantee a system of higher education 
fully integrated at European level, namely in terms of quality, levels of 
participation and employability of graduates; 

• the Government’s commitment to support efforts to protect and improve the 
quality of academic teaching and learning, in a way that will foster participation 
in international networks of higher education and the full recognition of 
Portuguese higher education institutions and degrees; 


• the Government’s acknowledgement that the challenges of maintaining quality, 
responsiveness and competitiveness in higher education are a major priority 
against the background of unprecedented levels of expansion, change and 
diversification in the sector in Portugal; 

• the Government’s strategy of promoting a diversified system of higher education, 
coping with heterogeneous publics and able to enlarge the overall number of 
students in Portugal, in a way that will foster quality at the various types of 
institutions (i.e., both polytechnics’ and universities); 

• the Government’s commitment to acknowledge the increasing importance of 
research, development and innovation for the knowledge society, to further 
support research and development in the higher education sector to achieve the 
highest international standards and to promote the transfer of outcomes of this 
research aiming at better supporting social, cultural and economic progress and 
maintaining the integral connection between research and teaching and their 
balanced development in higher education institutions. 

• the Government’s commitment to develop quality assurance procedures and 
processes, in line with best international practice and to facilitate the conduct of 
reviews of the effectiveness of quality assurance agencies, their procedures and 
their outcomes; 

• the Government’s commitment to increase the access of new publics to higher 
education and to increase the provision of lifelong education. 

LEGISLATIVE BACKGROUND 

The Government, through the MGTES, establishes and implements procedures for 
quality assurance, and arranges for a review of the effectiveness of these procedures, 
under the evaluation law (Law n. 38/94, 21 November, followed by Decree-Law n. 
205/98, of 11 July and Decree-Law n. 88/2001, of 23 March) and/or other legislative 
instruments in force at the time of the review. 

The National Council for the Evaluation of Higher Education, GNAVES, has a statutory 
function (Decree-Law n. 205/98, 11 July) to assist and assure the credibility of the 
process of higher education evaluation, and to review and report on the quality 
assurance procedures. 

The evaluation law (Law n. 38/94, 21 November) determines that the evaluation of 
higher education includes procedures of self- and external-evaluation, with the latter 
attributed to “Representative Institutions, which include: 

• The Eoundation of Portuguese Universities (EUP) 

• The Portuguese Association of Poly technic Institutes (ADISPOR) 

• The Portuguese Association of Private Higher Education (APESP) 

In addition, these procedures should be complemented by an overall evaluation of the 
system of higher education (Law n. 38/94, 21 November; Article 9, n. 3), which has 
never been made. 

The accreditation of the higher education system has been attributed (through 
Law 1/2003, of 6 January; article 36, n.3) to the Institutions also in charge of the 


^ In Portugal the term Polytechnics is used to characterise what is generally known as non-university higher education institutions. 


evaluation process, but its practice has not been regulated and, therefore could not 
be implemented. As a result, the accreditation of the higher education system needs 
to be designed, regulated and implemented following current trends at European and 
international levels. 

OBJECTIVES 

The exercise will examine the following topics: 

• a review of the available accreditation and quality assurance practices; 

• a review of the Portuguese National Council for the Evaluation of Higher 
Education, CNAVES, and the related activities and procedures of “Representative 
Institutions”; 

• a review of the available professional accreditation practices; 

• recommendations that will guarantee the establishment of a national 
accreditation system and practices which will meet the European Standards and 
Guidelines for Quality Assurance in the European Higher Education Area, EHE A, 
as adopted by ministers in Bergen in May 2005. 

This exercise should follow the European trends for an increased internationalisation 
of the quality assurance and accreditation approaches and agencies, as stated in the 
ENQA Report to the Ministers meeting in Bergen, “Standards and Guidelines for 
Quality Assurance in the European Higher Education Area” (2005). In particular, the 
ENQA recognises the need for a cyclical review of the national member agencies, and 
recommends: “any European agency should at no more than five-year intervals conduct 
or be submitted to a cyclical external review of its processes and activities”. 

TERMS OE REEERENCE 

The European Association for Quality Assurance in Higher Education (ENQA) will appoint 
an International Review Team (IRT) to review the present Portuguese National Gouncil 
for the Evaluation of Higher Education, CNAVES, and the related activities and 
procedures of “Representative Institutions” (FUP, ADISPOR, APESP). The review team 
appointed by ENQA will be responsible for preparing recommendations aiming at 
meeting the standards and guidelines for quality assurance in the European Higher 
Education Area, EHEA. 

The review team will evaluate the quality assurance practices coordinated by 
CNAVES and will report on their effectiveness. The MCTES will provide other relevant 
statistical information on the Portuguese higher education system. 

The main objectives of this task are: 

• to review the accreditation and quality assurance system for Portuguese higher 
education and related practices. 

• to advice the CNAVES and the MCTES on appropriate academic and management 
structures for implementing adequate quality assurance and accreditation 
practices. 

• to promotte the idea of quality as a core value that is integrated with the principal 
research, teaching and administrative activities of higher education institutions. 

• to provide recommendations for fulfilling the European standards for quality/ 
accreditation agencies in the EHEA. 


Guidelines and obligations 

The CNAVES and the “Representative Institutions” (FUP, ADISPOR, APESP) will 
cooperate with MCTES in the evaluation exercise and will prepare self-evaluation 
reports. The ENQA will provide a set of guidelines for use by the CNAVES and the 
“Representative Institutions” in order to allow the preparation of their self-evaluation 
reports and the other parts of the review process. 

Outcomes: 

Draft Report 

The ENQA, through the responsibility of the review team, will provide the CNAVES 
and the “Representative Institutions” (EUP, ADISPOR and APESP) with a draft report 
on the findings of the review. ENQA will specify a period for the CNAVES and the 
“Representative Institutions” to comment on its contents. 

Final Report 

The ENQA, through the responsibility of the review team, will provide the MCTES and 
the CNAVES with a final report on the findings of the review. The report will also 
include recommendations for improvement and for achieving the European standards 
for quality/accreditation agencies. 

The CNAVES will make the final report publicly available and will organise its public 
debate. 


Appendix II 

ENQA panel 

Christian Thune was for a number of years Professor in International Politics at the 
University of Copenhagen, and he was Dean of the Faculty of Economics, Law and 
Political Science when in 1989 he was appointed by the Danish government Director 
of the new Centre for Quality Assurance and Evaluation of Higher Education. 

During the period of 1992-99 the Centre evaluated all higher education programmes 
in Denmark. In 1999 Christian Thune was appointed Executive Director of the new 
Danish Evaluation Institute that was established by the government with the mandate 
systematically to evaluate all levels and sectors of Danish education. 

From 2000-2005 Christian Thune was President of the European Association for 
Quality Assurance in Higher Education (ENQA). He was from 1997-99 3 ^ director 
of the International Network for Quality Assurance Agencies in Higher Education 
(INQAAHE). He was a member 1994-2003 of the Hong Kong Council of Academic 
Accreditation. 

In 2000 Christian Thune was a member of the panel of experts who reviewed the 
Hungarian Accreditation Council (HAC) and he is now a member of the international 
advisory board of HAC. Christian Thune is also a member of the Quality Assessment 
Commission of the Catalan Quality Assurance Agency (AQU), and of the board of the 
Evaluation and Accreditation Organisation of Lower Saxony, Germany (ZEWA). 

David Dill is Professor of Public Policy at the University of North Carolina at Chapel 
Hill and Director of the Ford-Foundation funded Research Program on Public Policy 
for Academic Quality (www.unc.edu/ppaq). From 1984-94 Professor Dill also served 
as Assistant to the Chancellor for Planning at UNC-CH. 

Professor Dill has been a Visiting Research Fellow at the University of Manchester 
Business School, a Visiting Fellow at Wolfson College, Cambridge University, a Visiting 
Professor at the Center for Higher Education Policy Studies (CHEPS) at the University 
of Twente in the Netherlands, and a Visiting Lecturer at the Norwegian Institute for 
Studies in Research and Higher Education (NIFU). He is a Past-President of the Society 
for College and University Planning in the US and a Fellow of The Society for Research 
in Higher Education in the UK. 

In 1999 Professor Dill was part of an international team that evaluated the 
University Grants Gommittee of Hong Kong’s Teaching-Learning Quality Process 
Reviews. Professor Dill has also been a consultant and/or evaluator on quality assurance 
to the Finnish Higher Education Evaluation Council (FINHEEC), the University Grants 
Gommittee in Hong Kong, the Hong Kong University for Science and Technology, the 
Gouncil of Universities in Ontario Ganada, the Swedish National Agency for Higher 
Education, and in the US to the Iowa Board of Regents, The University of Missouri 
System, and the Western Association of Schools and Golleges Accreditation Agency. 

Helmut Konrad has since 1984 been a full professor of History at the University of 
Graz and he has received a number of academic honours and held important positions 
in the field. He was from 1987 dean of the Faculty of Humanities. From 1993-97 he 


went on to serve as Rector of the university and after his period as Rector he continued 
till 1999 as Pro-rector. From 1995-98 he was Vice-president of the Austrian Rectors’ 
Conference. 

Helmut Konrad was from 2000-2005 President of the Austrian Accreditation 
Council and he has since 1999 been a member of the German Accreditation Council. 

From 2004-2005 Helmut Konrad was a member of the ENQA Board. 

Gemma Rauret has developed her academic career at the University of Barcelona, 
in the Analytical Chemistry Department as well as in the Faculty of Chemistry. She 
became a full professor in 1984. She has been Dean of the Faculty of Chemistry and 
General Secretary of the University of Barcelona. 

In 1998 she was appointed Director of the Quality Assurance Agency for the 
University System in Catalonia (AQU Catalunya) by the University Minister of 
Catalonia and President of the Quality Assessment Commission of AQU. As Director 
of AQU Catalunya she has been very active in promoting quality assessment in the 
Catalan university system. She has also participated in the National Plan for university 
assessment of the Spanish Ministry of Education as a member of the Technical 
Committee. 

In 2006, she was appointed Director of ANECA, the national Spanish agency for 
evaluation and accreditation. 

Gemma Rauret has been involved in various international assessment projects, 
including the TEEP Project I and II (Trans European Evaluation Project, ENQA) 
devoted to build up and test international assessment methodology for Bachelor and 
Joint Master’s programmes. She was a member of the ENQA working group that did 
the initial drafting of and discussions on the European standards for quality assurance 
agencies of higher education. 

Anette Dorge Jessen has her academic background in administration and 
international development studies. 

She has extensive experience with evaluation and quality assurance gained from 
different professional contexts. She worked from 1994-1996 as desk officer in the Danish 
Ministry for Eoreign Affairs with responsibility for the evaluation of projects funded 
under the Danish governmental aid and development programme. Erom 1997-2001 
she worked for one of the major Danish private aid organisations, first as programme 
coordinator for the programmes in Asia and from 1999 as programme director with the 
overall responsibility for the strategic development and quality assurance of the entire 
programme portfolio of the organisation. 

Since 2001 Anette Dorge Jessen has worked in the Danish Evaluation Institute, first 
as evaluation officer and from 2005 as director of projects with responsibility for the 
department for quality assurance and accreditation of higher education. She has been 
responsible for several major projects, including an international comparative review of 
agricultural science programmes in Denmark, the Netherlands, Germany and Ireland. 

She is the deputy chairman of the board of the Danish Evaluation Society. 


Appendix III 

Professional associations 

According to the terms of reference the ENQA review should take into account the 
existing accreditation practices regarding professional educations. In order to document 
these activities the ENQA panel invited a number of professional associations in 
Portugal to submit a brief account on their current accreditation practices. 

Six associations accepted the invitation and submitted an account: 

• The Portuguese Association of Technical Engineers (ANET) 

• The Portuguese Order of Engineers 

• The Portuguese Veterinary College 

• The Portuguese Order of Architects 

• The Portuguese Order of Nurses 

• Portuguese Pharmaceutical Society 

The ENQA panel interviewed the six associations during the site visit in Lisbon in 
May, 2006 in order to further elaborate the panel’s understanding of the associations’ 
practices and views on a future quality assurance system. 
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Appendix IV 

Site visit programme 

MONDAY, 15 MAY 


TIMESLOT 

ACTIVITY 

PANEL PARTICIPANTS 

PLACE 

8:00 - 9:15 am 

Panel meeting 

All panel members 

Hotel Marriott 

9:30 - 11:00 am 

Meeting with the Director General 
for Higher Education, Antonio Morao 
Dias, and experts from the Ministry 
of Science, Technology and Higher 
Education (Afonso Costa, Teresa 
Patricio) 

All panel members 

Ministry of Science, 
Technology and 
Higher Education 

11:00 am - 12:30 am 

Meeting with Jose Mariano Gago, 
Minister of Science, Technology and 
Higher Education and Manuel Heitor, 
State Secretary of Science, 
Technology and Higher Education 

All panel members 

MCTES 

13:00 - 14:30 pm 

Lunch with Luis Magalhaes, 
President of Knowledge Agency, 
UMIC, (Former President of Science 
Foundation and coordinator of first 
evaluation panel of R&D centres in 
1996) 

All panel members 

Hotel Marriott 

14:30 - 16:00 pm 

Meeting with the CNAVES 
management (Adriano Moreira, 
President) and the self-evaluation 
group - Sergio Machado dos Santos 

All panel members 

Hotel Marriott 

16:00 - 17:30 

Evaluation Council for Public 
Universities (FUP) 

Christian Thune, 
Gemma Rauret and 
Anette Dorge lessen 

Hotel Marriott 


Evaluation Council for Public 
Polytechnic (ADISPOR) 

David Dill, Helmut 
Konrad and Brit 
Buchhave 

Hotel Marriott 

17:30 - 19:00 

Evaluation Council for Private 
Universities (APESP) 

Christian Thune, 
Gemma Rauret and 
Anette Dorge lessen 

Hotel Marriott 


Evaluation Council for Private 
Polytechnic (APESP) 

David Dill, Helmut 
Konrad and Brit 
Buchhave 

Hotel Marriott 

20:00 

Dinner with President Joao 
Sentieiro and Vice-President Ligia 
Amancio of the Science and 
Technology Foundation 

All panel members 

Hotel Marriott 
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TUESDAY, 16 MAY 


TIMESLOT 

ACTIVITY 

PANEL PARTICIPANTS 

PLACE 

9:00 - 11:00 am 

Meeting with Representative 
Institutions for Public Universities 
(FUP) 

Chair Christian Thune, 
Gemma Rauret, 

Anette Dorge lessen 

Hotel Marriott 


Meeting with Representative 
Institutions for Private Universities 
(APESP) 

Co-chair David Dill, 
Helmut Konrad, 

Brit Buchhave 

Hotel Marriott 

11:00 am - 13:00 pm 

Meeting with Representative 
Institutions for Public Polytechnics 
(ADISPOR) 

Co-chair David Dill, 
Helmut Konrad, 

Brit Buchhave 

Hotel Marriott 


Meeting with student associations: 
AAL 

FPolytechnics 
FAIRe 
FA Porto 

Chair Christian Thune, 
Gemma Rauret, 

Anette Dorge lessen 

Hotel Marriott 

13:30 -14:45 pm 

Lunch with Eduardo Mar^al Grilo, 
Member Administration Calouste 
Gulbenkian Foundation; Former 
Minister of Education 

All panel members 

Gulbenkian 

Foundation 

15:00 - 17:00 pm 

Meeting with the National 
Committee of the Bologna Process: 
lose Luis Cardoso, Sebastiao Feyo 
Azevedo, Joao Francisco, Jorge 
Carvalhal, Alberto Amaral, 

Bruno Carapinha 

All panel members 

DGES 

17:00 - 18:30 pm 

Meeting with CRUP - Council of 
Rectors 

All panel members 

DGES 

20:00 

Dinner with Manuel Heitor, 
Secretary of State for Science, 
Technology and Higher Education 
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WEDNESDAY, 17 MAY 


TIMESLOT 

ACTIVITY 

PANEL PARTICIPANTS 

PLACE 

9:00 - 11:00 am 

Meeting with representatives from 
one public university: 

Institute Superior Economia e Gestao 
Universidade Tecnica de Lisboa 

Chair Christian Thune, 
Gemma Rauret, 

Anette Dorge Jessen 

Universidade 
Tecnica de Lisboa 
(ISEG) 


Meeting with representatives from 
one public polytechnic: 

Politecnico de Lisboa 

Co-chair David Dill, 
Helmut Konrad, 

Brit Buchhave 

Politecnico de Lisboa 

11:15 “ 13:00 pm 

Meeting with representatives from 
one private university: 
Universidade Lusiada 

Chair Christian Thune, 
Gemma Rauret, 

Anette Dorge Jessen 

Universidade 

Lusiada 


Meeting with representatives from 
one private polytechnic: 

Escola Superior de Educa^ao Joao 
de Deus 

Co-chair David Dill, 
Helmut Konrad, 

Brit Buchhave 

Escola Superior de 
Educa^ao Joao de 
Deus 

13:30 - 15:00 pm 

Lunch with President of the 
Coordinating Council of S&T, 
Joao Lobo Antunes 

All panel members 

Hotel Marriott 

15:30 - 17:00 pm 

Meeting with legal expert 
Paz Ferreira 

All panel members 

Hotel Marriott 

17:00 - 19:00 pm 

Meeting The Architects’ Professional 
Association 

All panel members 

Hotel Marriott 

20:00 

Dinner with the Minister of Science, 
Technology and Higher Education 

All panel members 
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THURSDAY, 18 MAY 


TIMESLOT 

ACTIVITY 

PANEL PARTICIPANTS 

PLACE 

9:00 - 11:00 am 

Meeting with professional association 
for Engineering (Ordem dos 
Engenheiros) 

Chair Christian Thune, 
Gemma Rauret, 

Anette Dorge Jessen 

Hotel Marriott 


Meeting with Veterinarians 
Professional Association 

Co-chair David Dill, 
Helmut Konrad, 

Brit Buchhave 

Hotel Marriott 

11:00 am - 13:00 pm 

Meeting with the Nurses’ 
Professional Association 
Meeting with Pharmacists’ 
Professional Association 

Chair Christian Thune, 
Gemma Rauret, 

Anette Dorge Jessen 

Hotel Marriott 


Meeting with the Technical 
Engineers’ Professional Association 
(ANET) 

Co-chair David Dill, 
Helmut Konrad, 

Brit Buchhave 

Hotel Marriott 

13:00 - 15:00 pm 

Lunch with National Council of 

Education 

Jiilio Pedrosa 

All panel members 

Hotel Marriott 

15:00 - 16:00 pm 

Meeting with teachers unions. 
Fenprof 

Chair Christian Thune, 
Helmut Konrad, 

Anette Dorge Jessen 

Hotel Marriott 


Meeting with teachers unions. 
SNESup 

Co-chair David Dill, 
Gemma Rauret, 

Brit Buchhave 

Hotel Marriott 

16:00 - 17:00 pm 

Meeting with Employers Association: 
CIP 

All panel members 

Hotel Marriott 

18:00 - 19:00 pm 

State Secretary for Employment and 
Professional Training 

All panel members 

Ministry of Labour 

20:00 

Dinner with Council of Associate 
Labs, and directors of main research 
centres 

All panel members 

Hotel Marriott 
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FRIDAY, 19 MAY 


TIMESLOT 

ACTIVITY 

PANEL PARTICIPANTS 

PLACE 

9:00 - 10:30 am 

Meeting with the State Secretary for 
Education Jorge Pedreira 

All panel members 

Ministry of 
Education 

11:00 am - 13:00 pm 

Meeting with the Director General 
for Higher Education and team 

All panel members 

DGES 

13:30 - 14:30 pm 

Lunch at the DGES 

All panel members 

DGES 

15:00 - 17:00 pm 

Meeting with the Minister of 
Science, Technology and Higher 
Education Jose Mariano Gago 

Chair Christian Thune, 
David Dill, 

Anette Dorge Jessen 
and Brit Buchhave 

MCTES 

17:30 - 18:30 pm 

Wrap-up meeting 

Chair Christian Thune, 
David Dill, 

Anette Dorge Jessen 
and Brit Buchhave 

Hotel Marriott 

20:30 

Dinner with the OECD review 
committee: Abrar Hasan, Daniel 
O’Hare, Jane Wellmann, Jon File, 
Michael Gallagher, Paavo Lopponen 

Chair Christian Thune, 
David Dill, 

Anette Dorge Jessen 
and Brit Buchhave 

Hotel Marriott 


Appendix V 

Summary list of European standards for quality assurance 

This appendix lists the European standards for quality assurance in higher education. 

It omits the accompanying guidelines. The guidelines can be found in ENQA’s report 
“Standards and Guidelines for Quality Assurance in the European Higher Education 
Area” (2005) at www.enqa.eu. 

The standards are in three parts covering internal quality assurance of higher 
education institutions, external quality assurance of higher education, and quality 
assurance of external quality assurance agencies. 

Part 1: European standards and guidelines for internal quality assurance 
within higher education institutions 

1.1 POLICY AND PROCEDURES FOR QUALITY ASSURANCE 

Institutions should have a policy and associated procedures for the assurance of the 
quality and standards of their programmes and awards. They should also commit 
themselves explicitly to the development of a culture which recognises the importance 
of quality, and quality assurance, in their work. To achieve this, institutions should 
develop and implement a strategy for the continuous enhancement of quality. The 
strategy, policy and procedures should have a formal status and be publicly available. 
They should also include a role for students and other stakeholders. 

1.2 APPROVAL, MONITORING AND PERIODIC REVIEW OF PROGRAMMES 
AND AWARDS 

Institutions should have formal mechanisms for the approval, periodic review and 
monitoring of their programmes and awards. 

1.3 ASSESSMENT OF STUDENTS 

Students should be assessed using published criteria, regulations and procedures which 
are applied consistently. 

1.4 QUALITY ASSURANCE OF TEACHING STAFF 

Institutions should have ways of satisfying themselves that staff involved with the 
teaching of students are qualified and competent to do so. They should be available to 
those undertaking external reviews, and commented upon in reports. 

1.5 LEARNING RESOURCES AND STUDENT SUPPORT: 

Institutions should ensure that the resources available for the support of student 
learning are adequate and appropriate for each programme offered. 

1.6 INFORMATION SYSTEMS 

Institutions should ensure that they collect, analyse and use relevant information for 
the effective management of their programmes of study and other activities. 


1.7 PUBLIC INFORMATION 

Institutions should regularly publish up to date, impartial and objective information, 
both quantitative and qualitative, about the programmes and awards they are offering. 

Part 2: European standards for the external quality assurance of higher 
education 

2.1 USE OF INTERNAL QUALITY ASSURANCE PROCEDURES 

External quality assurance procedures should take into account the effectiveness of the 
internal quality assurance processes described in Part i of the European Standards and 
Guidelines. 

2.2 DEVELOPMENT OF EXTERNAL QUALITY ASSURANCE PROCESSES 

The aims and objectives of quality assurance processes should be determined before 
the processes themselves are developed, by all those responsible (including higher 
education institutions) and should be published with a description of the procedures 
to be used. 

2.3 CRITERIA FOR DECISIONS 

Any formal decisions made as a result of an external quality assurance activity should 
be based on explicit published criteria that are applied consistently. 

2.4 PROCESSES FIT FOR PURPOSE 

All external quality assurance processes should be designed specifically to ensure their 
fitness to achieve the aims and objectives set for them. 

2.5 REPORTING 

Reports should be published and should be written in a style, which is clear 
and readily accessible to its intended readership. Any decisions, commendations or 
recommendations contained in reports should be easy for a reader to find. 

2.6 FOLLOW-UP PROCEDURES 

Quality assurance processes which contain recommendations for action or which 
require a subsequent action plan, should have a predetermined follow-up procedure 
which is implemented consistently. 

2.7 PERIODIC REVIEWS 

External quality assurance of institutions and/or programmes should be undertaken on 
a cyclical basis. The length of the cycle and the review procedures to be used should be 
clearly defined and published in advance. 


2.8 SYSTEM-WIDE ANALYSES 

Quality assurance agencies should produce from time to time summary reports 
describing and analysing the general findings of their reviews, evaluations, assessments 
etc. 

Part 3: European standards for external quality assurance agencies 

3.1 USE OE EXTERNAL QUALITY ASSURANCE PROCEDURES EOR HIGHER 
EDUCATION 

The external quality assurance of agencies should take into account the presence and 
effectiveness of the external quality assurance processes described in Part 2 of the 
European Standards and Guidelines. 

3.2 OEEICIAL STATUS 

Agencies should be formally recognised by competent public authorities in the 
European Higher Education Area as agencies with responsibilities for external quality 
assurance and should have an established legal basis. They should comply with any 
requirements of the legislative jurisdictions within which they operate. 

3.3 ACTIVITIES 

Agencies should undertake external quality assurance activities (at institutional or 
programme level) on a regular basis. 

3.4 RESOURCES 

Agencies should have adequate and proportional resources, both human and financial, 
to enable them to organise and run their external quality assurance process(es) in an 
effective and efficient manner, with appropriate provision for the development of their 
processes and procedures. 

3.5 MISSION STATEMENT 

Agencies should have clear and explicit goals and objectives for their work, contained 
in a publicly available statement. 

3.6 INDEPENDENCE 

Agencies should be independent to the extent both that they have autonomous 
responsibility for their operations and that the conclusions and recommendations 
made in their reports cannot be influenced by third parties such as higher education 
institutions, ministries or other stakeholders. 

3.7 EXTERNAL QUALITY ASSURANCE CRITERIA AND PROCESSES USED BY 
THE AGENCIES 

The processes, criteria and procedures used by agencies should be pre-deflned and 
publicly available. These processes will normally be expected to include: 


• a self-assessment or equivalent procedure by the subject of the quality assurance 
process; 

• an external assessment by a group of experts, including, as appropriate, (a) student 
member(s), and site visits as decided by the agency; 

• publication of a report, including any decisions, recommendations or other formal 
outcomes; 

• a follow-up procedure to review actions taken by the subject of the quality 
assurance process in the light of any recommendations contained in the report. 

3.8 ACCOUNTABILITY PROCEDURES 

Agencies should have in place procedures for their own accountability. 


Appendix VI 

Overview of recommendations 

Quality assurance terminology 

• The ENQA panel recommends that the name of the new agency reflects the 
recommendation below that the agency covers accreditation and evaluation/audit. 
The panel’s specific recommendation bears the title “The Quality Assurance 
Agency” and not “The Accreditation Agency”. 

• The ENQA panel recommends that the new agency adopts the following 
definition of accreditation: The characteristic of accreditation is that the process 
is based on established external criteria and that the result of the process is a 
decision as to whether these criteria have in fact been met by the institution or 
programme under accreditation 

• The ENQA panel recommends that academic audit be defined by the new agency 
as a method for evaluating the strengths and weaknesses of the quality assurance 
frameworks and mechanisms adopted by a higher education institution for its own 
use in order to continuously monitor and improve its activities and results. 

National versus international accreditation 

• The ENQA panel recommends that the new agency covers all Portuguese study 
programmes of higher education and, accordingly, that the accreditations carried 
out by the national professional associations do not need to continue. 

• The ENQA panel recommends that higher education institutions and programmes 
have the formal right to let themselves be accredited by international accreditors, 
but on a self-funding basis and without the option of excusing themselves from the 
mandatory accreditations of the new Portuguese agency. 

• The ENQA panel recommends that the board of the quality assurance agency has 
the mandate to accept or refuse reports from other national or European operators 
and to make all accreditation decisions irrespective of operator responsible for 
the process 

Methods and levels of a Portuguese quality assurance system 

• The ENQA panel recommends that accreditation at programme level be combined 
with academic audit at the level of higher education institutions. 

• The ENQA panel recommends that the new quality assurance agency decides on 
a first cycle of accreditations at the programme level, beginning in the second 
half of 2007; 

• The ENQA panel recommends that this first cycle covers at least the BA level 
of study programmes in all public and private university and polytechnic study 
programmes; 

• The ENQA panel recommends that the new quality assurance agency announces 
a future cycle of academic audits and then carefully designs an academic audit 
process appropriate to the needs of Portuguese higher education to be introduced 
following the completion of the first cycle of programme accreditations; 


• The ENQA panel recommends that the new agency consider introducing 
institutional accreditation in order to steer and regulate the number and quality of 
the higher education institutions. 

One strong national agency 

• The ENQA panel recommends establishing an agency with clear responsibilities 
for quality assurance and accreditation. . An independent professional board 
appointed by government with decision-making authority should guarantee the 
independence of the organization. In addition, an advisory council should be 
established with a broader representation of relevant stakeholders. 

Independency of the agency 

• The ENQA panel recommends that the operational independence of a new quality 
assurance organisation in Portugal be ensured through a legal framework that 
clearly describes the mandate and tasks of the quality assurance agency, while 
providing the necessary flexibility for the operational level to make necessary 
adjustments to operational procedures. 

Data and information system 

• The ENQA panel recommends that an early project of the new agency be to ensure 
that information for each study programme needs to be publicly available and to 
make recommendations to the Ministry regarding the best means of providing 
these data 

Cross-sector reviews 

• The ENQA panel recommends that the new agency establish a procedure for 
producing summary reports every second or third year. 

Professionalism and consistency in the work of the quality assurance agency 

• The ENQA panel recommends establishing an agency with a strong and highly 
qualified staff and with a clear division of responsibilities between external expert 
panels and the staff 

• The ENQA panel recommends that an executive director is appointed with a 
strong personal profile and relevant experiences in organisation building and 
quality assurance. 

• The ENQA panel recommends that the quality assurance agency be established on 
the basis of a budget which provides for a professional staff consisting of not less 
than twenty-five qualified members. 

• The ENQA panel recommends that in the establishment phase the quality 
assurance agency cooperates with external consultants with international 
experience to advice on organisational and methodological matters. 

Use and role of experts 

• The ENQA panel recommends that the quality assurance agency establishes and 
enforces a no-conflict-of-interest mechanism in the appointment and work of its 
external experts. 


• The ENQA panel recommends that experts are appointed through a process that 
ensures a balanced age and gender profile and an appropriate insight into quality 
assurance. 

• The ENQA panel recommends that the new quality assurance agency receives 
the necessary funding to organise systematic training of experts for the review 
panels. 

Use of international experts 

• The ENQA panel recommends that members of external review panels typically 
number four-five members. Of these, at least two members must come from 
countries and or higher education institutions outside Portugal. At least one 
member must be from Portugal, but not represent a Portuguese higher education 
institution. 

• The ENQA panel recommends that the funding of the new quality assurance 
agency is at a level which makes it feasible to conduct reviews in English or 
another international language. 

Inclusion of students in the quality assurance process 

• The ENQA panel recommends that the new quality assurance agency firstly 
ensures that it is an unconditional element in the self evaluation processes and 
the site visits that student representatives have a roll that can lead to actual 
contribution to the process. Secondly, the new agency should set up procedures 
that ensure that appropriateness of including students is considered when external 
review panels are composed. 

• The ENQA panel recommends that the Portuguese government appoint 
representatives of student organisations to the advisory council. 

Stakeholder involvement 

• The ENQA panel recommends that the view of relevant stakeholders, e.g. 
employers and graduates, be taken into account in external quality assurance 
processes. 

International cooperation and networking 

• The ENQA panel recommends that the new quality assurance agency be given 
the directive and the means to engage actively at all levels of the agency in 
international and European cooperation and networking 

Internal accountability procedures and compliance with European standards 

• The ENQA panel recommends that a mid-term review of the new agency takes 
place by the end of 2009. 

• The ENQA panel recommends that the new agency gives a high priority to a 
process leading towards full compliance with all the European standards and 
thus secures its placement in the coming European register of external quality 
assurance agencies at the very latest by the end of 2012. 


Pre-accreditation, accreditation and relevance 

• The ENQA panel recommends introducing a two-step accreditation process of 
pre-accreditation and ex-post accreditation in order to create a high degree of 
transparency and consistency in the two processes; 

• The ENQA panel recommends that the final decision for provision of new study 
programmes be vested within the Ministry based on criteria for relevance and 
national concern. 

Financing of the new quality assurance agency 

• The ENQA panel recommends that a principle for co-financing is followed 
using a size parameter. The degree of variation according to the size-parameter 
between the highest and lowest possible contribution from the higher education 
institutions could then be settled between a 25 and 50 percent contribution. 

The focus of the accreditation criteria 

• The ENQA panel recommends that programme accreditations be based on generic 
as well as programme specific criteria in order to provide a comprehensive picture 
of the quality of the study programmes; 

• The ENQA panel recommends that differentiated criteria are formulated to reflect 
the characteristics of study programmes offered at universities and polytechnics, 
respectively; 

• The ENQA panel recommends that the application of the European Qualification 
Framework in the definition of generic cycle-based criteria; 

• The ENQA panel recommends that the area specific criteria cover input, 
processes and learning outcome that comply with the European standards, and 
when relevant take into account criteria applied internationally by professional 
associations in the respective fields. 

The process for defining the accreditation criteria 

• The ENQA panel recommends that the criteria be formulated through appropriate 
interaction between the legislative/framework-setting and the operational agency 
level, and that this process guarantees sufficient independence in relation to the 
education institutions involved; 

• The ENQA panel recommends that the criteria for professional programmes be 
developed in close cooperation with the professional associations, taking into 
account the criteria that already exist. 

Categories of accreditation decisions and follow-up 

• The ENQA panel recommends that an accreditation system includes all three 
categories of decision, and that the credibility of the agency is consciously 
supported by the use of all three categories. 

• The ENQA panel recommends that the quality assurance agency carries the 
responsibility for follow-up on conditional accreditation. 


Consequences of accreditation 

• The ENQA panel recommends that the consequences of accreditation are given a 
clear legal basis and that practical implications for higher education institutions 
are clearly spelled out by the new agency. 

• The ENQA panel recommends that the new accreditation processes are planned 
in such a way that higher education institutions are encouraged to respond to the 
criteria applied at the highest level appropriate. 

Length of accreditation cycle 

• The ENQA panel recommends that the length of the cycle be no less than five 
years, and no greater than six years. 

The need for an appeals form for accreditation decisions 

• The ENQA panel recommends that the formal framework and jurisdiction of an 
independent appeals forum be an element in a Portuguese accreditation system. 
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Appendix VII 

Budget for initial costs of establishing the Quality Assurance Agency 


BUDGET ITEM 

AMOUNT IN EURO 

COMMENT 

Salary 

Operation costs 

Renovation of the office premises 


150 000,00 


Furniture 


75 000,00 


IT (network, servers, pc, printers, 
copiers, telephone system) 


100 000,00 


Development of procedures 
and methods 

80 000,00 



Marketing/Branding 

5 000,00 

40 000,00 

Expenditure to produce 
materials that can brand 
and market the institution, 
including time for 
presentations to explain 
the new role of the agency 

Recruitment 

15 000,00 

20 000,00 


Total 

100 000,00 

385 000,00 



485 000,00 
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Estimated annual budget for running costs 

Public and Private Polytechnics 


BUDGET ITEM 

AMOUNT IN EURO 

COMMENT 

Salary 

Operation costs 

6 Academic staff 

199 920,00 


Based on a average salary 
level of 33.320 

Fee for experts 


80 000,00 

32 experts with a fee each 
of 2500 

Travel in Portugal 


17 088,00 

32 experts x 534 for travel 
in Portugal 

International travel 


35 000,00 

Half of the experts are 
international = 16 experts 
X 1400 for international 
travel 

Accommodation 


64 000,00 

32 experts x 2000 
(tentative) 

Accreditation of new programmes 


100 000,00 

Price for each pre- 
accreditation (desk research' 
50 X 2000. To be balanced by 
a fee from the institutions 

(note 5) 

Production of reports 


76 000,00 

Price for each report: 
152x500 

Total 

199 920,00 

372 088,00 


572 008,00 


General assumptions: 

That the total number of study programmes are 759 divided by 532 public and 227 
private. The 759 study programmes are divided into 20 wider study field areas. The 
accreditation will be conducted within a five year cycle including four groups of study 
programmes per year with 38 programmes in each. New programmes applying for 
pre-accreditation are estimated to 50 per year. 


Notes: 

1) Salary level for lectures 

2) 8 experts in each groups of study fields. Each group covers 19 programmes. It is recommended to split the experts into two teams 
with four in each (2x4) to make the process manageable in terms of number of visits, etc. 

3) The level of fee for experts is based on the level in other countries (for instance Denmark and Holland) 

4) The estimated cost for an international flight to and from Portugal is 1400 EURO. 

5) The estimated cost of 2000 EURO for a pre-accreditation is based on the cost for a desk research i.e. without the involvement of 
external experts 

6) 152 reports with production costs of 500 EURO each 


Estimated annual budget for running costs 

Public and private universities (undergraduate programmes) 


BUDGET ITEM 

AMOUNT IN EURO 

COMMENT 

Salary 

Operation costs 

12 Academic staff 

399 840,00 


Based on a average salary 
level of 33.320 

Fee for experts 


160 000,00 

2x4 experts in each group 
= 64 experts with a fee each 
of 2500 

Travel in Portugal 


34 176,00 

64 experts x 534 for travel 
in Portugal 

International travel 


44 800,00 

Half of the experts are 
international = 32 experts 
X 1400 for international 
travel 

Accommodation 


128 000,00 

64 experts x 2000 
(tentative) 

Accreditation of new programmes 


100 000,00 

Price for each pre- 
accreditation (desk research' 
50 X 2000. To be balanced by 
a fee from the institutions 

(note 5) 

Production of reports 


100 000,00 

Price for each report: 
200 X 500 

Total 

399 840,00 

566 976,00 


966 816,00 


General assumptions 

That the total number of undergraduate programmes are 1004 divided by 647 public, 
299 private and 58 offered by the Catholic university. The 1004 study programmes 
are grouped into 40 larger fields of study with 25 study programmes in each. The 
accreditation will be conducted within a five year cycle including 8 groups of study 
fields per year with 25 programmes in each. New programmes applying for pre- 
accreditation are estimated to 50 per year. 

Notes: 

1) salary level for lectures 

2) 8 experts in each group of study fields. Each group covers 25 programmes. It is recommended to split the experts into two teams 
with four in each (2x4) in order to make the process managerable in terms of number of visits, etc. 

3) The level of fee for experts is based on the level in other countries (for instance Denmark and Holland) 

4) The estimated cost for an international flight to and from Portugal is 1400 EURO. 

5) The cost of 2000 EURO is based on the cost for a desk research i.e. without the involvement of external experts 

6) 200 reports with production costs of 500 EURO each 
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Common costs for the Quality Assurance Agency 


BUDGET ITEM 

AMOUNT IN EURO 

COMMENT 

Salary 

Operation costs 

1 Executive Director 

60 000,00 


(note 1) 

1 academic secretary to the 
executive director 

33 320,00 


(note 2) 

8 Administrative staff 

134400,00 


8 X 16 800 EURO 

House Rent 



(note 4) 

Utilities 



(note 5) 

Cleaning 


15 000,00 

(note 6) 

Office supplies 


8 000,00 

(note 7) 

Postage and courier service 


12 000,00 

(note 8) 

Telecommunication 


10 000,00 

(note 9) 

Maintainance of computer 
facilities etc. 


30 000,00 

(note 10) 

Staff Development 


50 000,00 

(note 11) 

International networks, 
conferences and seminars 


30 000,00 


Board 


10 000,00 

(note 12) 

Advisory Council 


10 000,00 

(note 13) 

Travel costs for staff 


50 000,00 

(note 14) 

Consultancy fees 


20 000,00 

(note 15) 

Cross-sector reports 
(production costs) 


20 000,00 

(note 16) 

Other expenditure 


10 000,00 

Unforseen running 
expenditure 

Total 

227 720,00 

275 000,00 



5027 

20,00 



Notes: 

1) Based on the salary level for a Director-General (Administrative Career) 

2) Based on the salary level for lectures 

3) Based on the salary level for Administrative Assistants 

4) House rent is not included in the budget following the recommendation from MCTES 

5) Expenditures for heating, water and electricity are difficult to assess. For MCTES to assess 

6) Based on an estimate compared to the salary level for administrative assistants 

7) Include furnitures, office supplies etc. 

8) Based on the rough estimate 

9) Includes expenditure for phone and internet services and is based on the experiences from the other countries 

10) The estimated costs are based on the experiences from other countries 

11) Include participation in courses and training seminars that support the development of staff competences. The cost for staff 

training is expected to decrease after a few years of establishment. 

12) Include costs for meetings and travel. Ten annual meetings 

13) Include costs for meetings and travel. Four annual meetings 

14) Travel costs for staff attending seminars, conferences etc. 

15) In the establishment phase the agency will benefit from external consultanties to advice on organisational and methodological 
matters 

16) The report recommends the production of summary reports describing and analysing the general findings of the reviews. It is 
anticipated that the agency should produce 1 report every year. The 20.000 EURO covers salary as well as operational costs. 


Total Annual Budget for the Quality Assurance Agency 


BUDGET ITEM 

AMOUNT IN EURO 


Salary 

Operation costs 

Polytechnics 

199 920,00 

372 088,00 


Universities 

399 840,00 

566 976,00 


Common costs 

227 720,00 

275 000,00 


Total annual budget 

827 480,00 

1 214064,00 



2 041 544,00 


Appendix VIII 

Criteria for accreditation of university study programmes and polytechnic 
study programmes 

This appendix lists an overview of relevant criteria that may be applied in programme 
accreditation. The criteria are differentiated by sub-sector and include criteria for 
respectively university and polytechnic study programmes. Accordingly, the criteria are 
designed to reflect the specific characteristics of both types of programmes. Concerning 
the university programmes the criteria are formulated to comply with the Bologna 
process that specifies the general outcome requirements for bachelor and master 
levels. Furthermore, the criteria emphasise the research profile and capacity of 
the programmes. Concerning the polytechnics the criteria include on the other 
hand knowledge and experience from the professional practice and the professional 
orientation of the programme. 

The two types of programmes also share many characteristics so that the criteria are: 

• Formulated to comply with national legal requirements; 

• Generic and function as requirements that apply to all types of study programmes 
within the sub-sector. The criteria are not area-specific. It is the responsibility 

of the study programme under accreditation and part of its self-assessment 
to describe the compatibility with area specific quality requirements. These if 
possible have been laid down in an (international) frame of reference; 

• Formulated to comply with the European standards and guidelines for internal 
quality assurance. 

Criteria for University Programmes 

A: Aims and organisation 

• The aims and organisation of the programme (e.g. bachelor and master) comply 
with relevant legalisation; 

• The aims of the programme are clearly formulated and made publicly available; 

• The aims are operationalised so as to allow an assessment of the extent to which 
they are fulfilled; 

• The aims are formulated and developed in consideration of the needs and 
requirements of the labour market broadly defined; 

B: Content and structure 

• The content of the programme is appropriate for the realization of the aims of 
the programme (e.g. combinations and prioritizations of courses and other study 
elements, the balance between compulsory and optional elements and between 
level and scope); 

• The programme complies with formal requirements for the size of the curriculum 

• Progression is evident in the different combinations and prioritizations of courses 
and elements in the programme; 

• The content of the programme includes both theoretical and practical elements; 


• The programme meets relevant needs and requirements of the labour market 
broadly defined; 

C: Competences 

• The programme has formulated output-competences for bachelor and master level 
that are comparable with the European Qualifications Framework/the Dublin 
Descriptors (and with a national framework, if developed); 

• The final qualifications correspond to the requirements made to a study 
programme in the relevant domain by other study programmes in Portugal and 
abroad (domain specific requirements) 

• The competences are expressed both in terms of academic contents and in terms 
of practical applications; 

• The competences clearly reflect academic and didactic principles that comply 
with the awarded degree and are realistic in relation to the duration of the 
programmes; 

D: Teaching methods 

• The teaching and learning methods reflect the aims of the programme; 

• The teaching and learning methods reflect academic and didactic principles, 
which support the realisation of the aims of the programme. 

E: Examination methods 

• The examination methods clearly reflect academic and didactic principles and 
sustain the realisation of the aims of the programme; 

• The examination methods are in accordance with the achievement of the intended 
competences of the programme; 

• The examination methods are coordinated across the whole programme to ensure 
that all competences and other learning outcomes are achieved. 

F: Teaching staff and research 

• Teachers’ academic, research and pedagogical qualifications reflect the aims of 
the programme; 

• Relevant research results and methods are continually presented to the students 
thereby sustaining the learning environment of the programmes; 

• The programme attracts qualified teachers and researchers and has a strategy 
which secures future recruitment; 

• Teaching is largely provided by academics and researchers who contribute to the 
development of the subject area; 

• The teaching given by the regular staff is complemented by lectures or case studies 
presented by external guest teachers (e.g. from the industry or other universities). 

G: Admission, graduation time and dropout rate 

• The terms of admission are relevant and sufficient to meet the academic 
requirements of the programme; 

• The programme can be successfully completed within the set time. 


• The programme systematically monitors graduation time and dropout rate and 
regularly compare these with those of related programmes to detect the effect of 
the strategy and possible new ways to enhance it; 

H: PhD studies 

• The level of recruitment of PhD students is appropriate to meet the needs for 
research in the future. 

I: Facilities 

• The IT-facilities are sufficient to fulfil the aims and intended competences of the 
programme; 

• The accommodation and material facilities support the development of teaching 
and research and are sufficient to implement the aims of the programmes. 

J: Employment and graduates 

• The programme systematically collects and uses feedback from graduates, 
employers, labour market representatives and other relevant organisations in the 
continuing development of the programmes; 

• The programme monitors the employment rate of graduates and regularly 
compares it with those of related programmes. 

K: Internationalisation 

• The programme has a strategy for internationalisation; 

• The internationalisation is reflected in the content and syllabus of the programme; 

• The programme offers opportunities for students to study abroad and for teaching 
staff to teach and conduct research activities abroad; 

• The programme endeavours to attract students from abroad and to facilitate the 
integration of these students; 

L: Quality assurance 

• The programme has implemented the university’s overall strategy for quality 
assurance in operational goals and procedures; 

• The responsibility for the quality assurance mechanisms is clearly allocated at the 
programme level; 

• The quality assurance mechanisms assure approval, monitoring and periodic 
reviews of the programme; 

• The student assessments are based on publicly available criteria, decisions and 
procedures that are applied consistently; 

• Procedures are in place to obtain knowledge of whether teachers are qualified and 
competent to teach. The results are available for external reviews; 

• The teaching is periodically and systematically evaluated. The evaluations should 
among other things cover mechanisms for receiving feedback from students ; 

• The resources available for the support of student learning are adequate and 
appropriate; 

• Relevant information for the effective management of the programme are 
collected, analysed and used; 


• Quantitative and qualitative information on the programme is published and is up 
to date, impartial and objective; 

• There is a systematic follow-up on the evaluations of both programme and 
teaching. The results of the evaluations and the plans for follow-up are available 
at the homepage of the university. 

Criteria for polytechnic programmes 

A: Aims and organisation 

• The aims and organisation of the programme comply with relevant legalisation; 

• The aims of the programme are clearly formulated and made publicly available; 

• The aims are operationalised so as to allow an assessment of the extent to which 
they are fulfilled; 

• The aims are formulated and developed in consideration of the needs and 
requirements of the labour market and the relevant profession; 

B: Content and Structure 

• The content and length of the programme are appropriate for the realization of the 
aims of the programmes (e.g. the combinations and prioritizations of courses and 
elements relate to the balance between compulsory and optional elements); 

• The programme complies with formal requirements for the size of the curriculum; 

• Progression is evident in the different combinations and prioritizations of courses 
and elements in the programmes; 

• The content includes both theoretical and practical elements in a close interaction 
with the professional field and/or applied research; 

• The programme includes practical training that is appropriate and relevant for 
the profession 

• The programme has clear links with current developments in the professional field 
and meets relevant needs and requirements of the labour market; 

C: Competences and professional orientation 

• The final qualifications of the programme correspond to the general, 
internationally accepted description of the qualifications of a bachelor (currently 
the Dublin decriptors) 

• The final qualifications and competences are partly based on the professional 
profiles and/or professional competences drawn up by or in conjunction with the 
relevant professional field 

• A bachelor from a polytechnic institution is qualified at the level of starting as a 
professional in a specific profession or related professional field for which a degree 
from a polytechnic institution is required 

• A bachelor from a polytechnic institution qualifies for a master programme 

D: Teaching methods 

• The teaching and learning methods reflect the aims of the programme; 

• Students acquire knowledge by studying professional literature, course materials 
that originate in the professional practice, and by interaction with the professional 
held and applied research 


E: Examination methods 

• The examination methods clearly reflect professional and didactic principles and 
sustain the realisation of the aims of the programme; 

• The examination methods are in accordance with the achievement of the intended 
competences of the programme; 

• The examination methods are coordinated across the whole programme to ensure 
that all competences and other learning outcomes are achieved. 

E: Teaching staff 

• Teachers’ qualifications reflect the aims of the programme; 

• The programme attracts qualified teachers and professionals and has a strategy 
which enhances future recruitment of qualified staff; 

• Teaching is largely provided by staff who relates the courses to the professional 
practice. 

• The teaching given by the regular staff is complemented by lectures or case 
studies presented by external guest teachers (e.g. from industry and other relevant 
professions). 

G: Admission, duration and dropout rate 

• The terms of admission are relevant and sufficient to meet the requirements of 
the programme; 

• The programme can be successfully completed within the set time. 

• The programme systematically monitors graduation time and dropout rate and 
regularly compare these with those of related programmes to detect the effect of 
the strategy and possible new ways to enhance it. 

H: Eacilities 

• The IT-facilities are sufficient to fulfil the aims and intended competences of the 
programme; 

• The accommodation and material facilities support the development of teaching 
and are sufficient to implement the aims of the programme. 

I: Employment and graduates 

• The programme systematically collects and uses feedback from graduates, 
employers, labour market representatives and other relevant organisations in the 
continuing development of the programme; 

• The programme monitors the employment rate of graduates and regularly 
compares it with those of related programme. 

J: Quality assurance 

• The programme has implemented the overall strategy of the institution for quality 
assurance in operational goals and procedures; 

• The responsibility for the quality assurance mechanisms is clearly allocated at the 
programme; 

• The quality assurance mechanisms assure approval, monitoring and periodic 
reviews of the programme; 


The student assessments are based on publicly available criteria, decisions and 
procedures that are applied consistently; 

Procedures are in place to obtain knowledge of whether teachers are qualified and 
competent to teach. The results are available for external reviews; 

The teaching is periodically and systematically evaluated. The evaluations should 
among other things cover mechanisms for receiving feedback from students; 

The resources available for the support of student learning are adequate and 
appropriate; 

Relevant information for the effective management of the programmes are 
collected, analysed and used; 

Quantitative and qualitative information on the programmes is published and is 
up to date, impartial and objective; 

There is a systematic follow-up on the evaluations of both programmes and 
teaching. The results of the evaluations and the plans for follow-up are available at 
the homepage of the polytechnic institution. 


